fix(websearch): sync tool_choice when converting web_search tools#31375
Conversation
Claude Code forces native web search via tool_choice pointing at web_search while websearch_interception renames the tool to litellm_web_search, causing Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced tool_choice to match the converted tool name. Co-authored-by: Cursor <[email protected]>
|
Shivam Rawat seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Greptile SummaryThis PR fixes two bugs on the websearch interception path for Anthropic
Confidence Score: 5/5Safe to merge — both fixes are narrowly scoped to the websearch interception path and do not touch non-websearch requests. The two code paths changed are guarded by the No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/integrations/websearch_interception/handler.py | Adds _tool_name and _sync_forced_tool_choice helpers; calls the latter inside async_pre_request_hook after tool conversion. Both methods are straightforward and well-tested. |
| litellm/llms/anthropic/experimental_pass_through/messages/handler.py | Forwards tool_choice explicitly into _execute_pre_request_hooks so it lands in hook kwargs and can be rewritten; extracts it back via request_kwargs.pop. Change is additive and doesn't affect callers that don't set tool_choice. |
| litellm/llms/custom_httpx/llm_http_handler.py | Extracts stream-re-wrapping logic into _maybe_websearch_fake_stream_wrap and applies it in both the post-agentic-plan path and the non-agentic fallback path. Double-wrapping is guarded by __aiter__ check; behavior for non-websearch requests is unchanged (flag not set → passthrough). |
| tests/test_litellm/integrations/websearch_interception/test_websearch_interception_handler.py | Adds five new unit tests covering _sync_forced_tool_choice (forced repoint, untouched, extra fields, non-forced passthrough) and an end-to-end hook test. All tests use mocks — no real network calls. |
| tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py | Adds four unit tests for _maybe_websearch_fake_stream_wrap covering the wrap, no-op (flag unset), no-op (already streaming), and no-op (None logging_obj) paths. All mock-based. |
Reviews (3): Last reviewed commit: "test(websearch): cover tool_choice sync ..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…lients When websearch interception converts stream=true to false for the agentic loop, dict responses from the loop were returned as application/json even though the client requested SSE. Wrap those responses in FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers (e.g. Claude Code) receive text/event-stream after search completes. Fixes #27721 Co-authored-by: Cursor <[email protected]>
…P006 Add regression tests for both websearch interception fixes: _sync_forced_tool_choice repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and _maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for streaming clients (#27721). Switch the new helper annotations to builtin dict/list so the ruff UP006 strict-rule ceiling stays within budget. Co-authored-by: Cursor <[email protected]>
|
@greptile-ai review again holistically. |
|
@greptile-apps review again holistically with all commits |
Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead. Co-authored-by: Cursor <[email protected]>
de82f78
into
litellm_internal_staging
Relevant issues
Fixes #27721
Fixes LIT-4039
Summary
Claude Code talking to LiteLLM proxy over /v1/messages with websearch_interception hit two bugs on the same path: search sub-requests 400'd before any search ran, and streaming clients got application/json back after search instead of SSE, which looked like truncated output. This PR fixes both so native web search works end-to-end through the proxy
Problem
Claude Code uses the Anthropic Messages API with a native web_search tool and often forces it via tool_choice: {"type": "tool", "name": "web_search"}. It also sends stream: true
LiteLLM websearch interception renames that tool to litellm_web_search and runs Tavily (or another configured search tool) inside an agentic loop. Two things broke on that path:
Search never started (HTTP 400)
Tools were renamed but tool_choice still pointed at web_search. Anthropic returned Tool 'web_search' not found in provided tools. Claude Code showed "Did 0 searches". A second gap made this worse: tool_choice is a named param on anthropic_messages(), not in **kwargs, so the websearch pre-request hook never saw it to rewrite it
Streaming clients got JSON after search ([Bug]: websearch_interception silently truncates streaming response on /v1/messages — follow-up call always uses stream=False #27721)
The pre-hook correctly converts stream: true → false for the agentic loop and sets websearch_interception_converted_stream so the response can be re-streamed later. After the agentic loop finished, the proxy returned a plain dict (content-type: application/json) instead of SSE. Clients expecting text/event-stream mis-parsed it and answers appeared cut off mid-generation with no error. FakeAnthropicMessagesStreamIterator only ran when the LLM didn't call the search tool; agentic-loop results bypassed the wrap entirely
Fix
Sync forced tool_choice with converted tools
Forward tool_choice explicitly into pre-request hooks. After converting web_search → litellm_web_search, repoint forced tool_choice via _sync_forced_tool_choice() so upstream Anthropic sees a consistent tool name
Re-wrap agentic loop responses as SSE for streaming clients
Add _maybe_websearch_fake_stream_wrap() and call it when the agentic plan returns and on the final handler return. If the client originally requested streaming (websearch_interception_converted_stream on the logging object) and the result is a dict, wrap it in FakeAnthropicMessagesStreamIterator so the proxy emits proper Anthropic SSE
Agentic follow-up calls stay non-streaming internally so LiteLLM can still detect chained tool_use across loop iterations; only the wire format back to the client is converted to SSE
Screenshots / Proof of Fix
Start proxy with websearch_interception and a Tavily search tool configured
Search runs (no 400):
Before: HTTP 400. After: HTTP 200 with web_search_tool_result and real URLs
Streaming wire format (#27721):
grep -i content-type /tmp/headers.txt
Before: content-type: application/json. After: content-type: text/event-stream with event: / data: SSE chunks
Files changed
litellm/integrations/websearch_interception/handler.py — _sync_forced_tool_choice()
litellm/llms/anthropic/experimental_pass_through/messages/handler.py — forward tool_choice into hooks
litellm/llms/custom_httpx/llm_http_handler.py — _maybe_websearch_fake_stream_wrap()