Repository navigation
fix: deepseek v4 inter-leaved tool and reasoning parser - #8901
Conversation
WalkthroughThis change removes special-case handling that disabled Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
93bba90 to
dcc5786
Compare
dcc5786 to
3abb3ba
Compare
3abb3ba to
e2d3487
Compare
|
Verified against DSV4 deployment - POST http://localhost:8000/v1/chat/completions model=deepseek-ai/DeepSeek-V4-Pro |
Overview:
Fixes DeepSeek V4 tool-continuation reasoning parsing without regressing Kimi K2.5. In the failing DSv4 flow, the request ends with a
role=toolmessage and the DSv4 formatter has already seeded<think>into the next assistant prompt. The model can then stream reasoning text followed by</think>without an opening<think>token in the completion. If the postprocessor clearsprompt_injected_reasoningfor all tool-continuation turns, the closing tag is treated as normal content and leaks to the client.This PR keeps DSv4 in injected-reasoning mode for that path, while preserving the Kimi K2.5 behavior introduced by
5671d8e374021ca1aeb399fa9729285903ea6645: direct post-tool final answers forkimi_k25should remain normalcontent, not be mislabeled asreasoning_content.Details:
last_is_toolsuppression that caused the DSv4</think>leak.kimi_k25, where the parser is force-reasoning and direct tool follow-up answers must bypass reasoning parsing.user -> assistant(tool_call) -> tool -> assistantwhere the completion starts inside reasoning and emits</think>.content.tool_choice=required/ named guarded from reasoning parsing for guided-decoding JSON outputs.Where should the reviewer start?
lib/llm/src/preprocessor.rs— parser-specific post-tool reasoning gate.lib/llm/tests/postprocessor_parsing_stream.rs— DSv4</think>leak regression and Kimi K2.5 non-regression coverage.lib/llm/tests/test_reasoning_parser.rs— parser-level DSv4 injected-reasoning tests.Validation:
Ran with
CUDARC_CUDA_VERSION=12090because local CUDA 13.2 is rejected bycudarc 0.19.3during build.cargo test -p dynamo-llm --no-default-features --test postprocessor_parsing_stream tool_continuation -- --nocapturecargo test -p dynamo-llm --no-default-features --test test_reasoning_parser deepseek_v4_post_tool -- --nocapturecargo test -p dynamo-llm --no-default-features --test test_reasoning_parser kimi_k25 -- --nocapturecargo test -p dynamo-llm --no-default-features --test tool_choice -- --nocapturecargo test -p dynamo-parsers _v4 -- --nocapturecargo test -p dynamo-parsers tool_calling::xml::kimi_k2_parser::tests -- --nocaptureRelated Issues:
5671d8e374021ca1aeb399fa9729285903ea6645