Skip to content

fix: deepseek v4 inter-leaved tool and reasoning parser - #8901

Merged
biswapanda merged 2 commits into
mainfrom
bis/dsv4-tool-plus-think
May 3, 2026
Merged

biswapanda merged 2 commits into
mainfrom
bis/dsv4-tool-plus-think

Conversation

@biswapanda

@biswapanda biswapanda commented Apr 30, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

Fixes DeepSeek V4 tool-continuation reasoning parsing without regressing Kimi K2.5. In the failing DSv4 flow, the request ends with a role=tool message and the DSv4 formatter has already seeded <think> into the next assistant prompt. The model can then stream reasoning text followed by </think> without an opening <think> token in the completion. If the postprocessor clears prompt_injected_reasoning for all tool-continuation turns, the closing tag is treated as normal content and leaks to the client.

This PR keeps DSv4 in injected-reasoning mode for that path, while preserving the Kimi K2.5 behavior introduced by 5671d8e374021ca1aeb399fa9729285903ea6645: direct post-tool final answers for kimi_k25 should remain normal content, not be mislabeled as reasoning_content.

Details:

  • Removes the broad last_is_tool suppression that caused the DSv4 </think> leak.
  • Reintroduces post-tool reasoning suppression specifically for kimi_k25, where the parser is force-reasoning and direct tool follow-up answers must bypass reasoning parsing.
  • Adds a DSv4 stream regression test for user -> assistant(tool_call) -> tool -> assistant where the completion starts inside reasoning and emits </think>.
  • Adds a Kimi K2.5 regression test for the same conversation shape where the final direct answer must stay in content.
  • Keeps tool_choice=required / named guarded from reasoning parsing for guided-decoding JSON outputs.

Where should the reviewer start?

  • lib/llm/src/preprocessor.rs — parser-specific post-tool reasoning gate.
  • lib/llm/tests/postprocessor_parsing_stream.rs — DSv4 </think> leak regression and Kimi K2.5 non-regression coverage.
  • lib/llm/tests/test_reasoning_parser.rs — parser-level DSv4 injected-reasoning tests.

Validation:

Ran with CUDARC_CUDA_VERSION=12090 because local CUDA 13.2 is rejected by cudarc 0.19.3 during build.

  • cargo test -p dynamo-llm --no-default-features --test postprocessor_parsing_stream tool_continuation -- --nocapture
  • cargo test -p dynamo-llm --no-default-features --test test_reasoning_parser deepseek_v4_post_tool -- --nocapture
  • cargo test -p dynamo-llm --no-default-features --test test_reasoning_parser kimi_k25 -- --nocapture
  • cargo test -p dynamo-llm --no-default-features --test tool_choice -- --nocapture
  • cargo test -p dynamo-parsers _v4 -- --nocapture
  • cargo test -p dynamo-parsers tool_calling::xml::kimi_k2_parser::tests -- --nocapture

Related Issues:


Open in Devin Review

@biswapanda
biswapanda requested a review from a team April 30, 2026 08:15
@biswapanda biswapanda self-assigned this Apr 30, 2026
@github-actions github-actions Bot added fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Apr 30, 2026
@coderabbitai

coderabbitai Bot commented Apr 30, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

This change removes special-case handling that disabled prompt_injected_reasoning when final chat messages have role tool, allowing the flag to flow through reasoning parser configuration. Corresponding regression tests validate DeepSeek V4 reasoning parsing and tool-continuation scenarios with reasoning boundaries.

Changes

Cohort / File(s) Summary
Production Code
lib/llm/src/preprocessor.rs
Removes special-case adjustment that disabled prompt_injected_reasoning for tool-role messages, allowing the flag to pass unmodified into reasoning parser configuration.
Test Additions
lib/llm/tests/postprocessor_parsing_stream.rs, lib/llm/tests/test_reasoning_parser.rs
Adds regression tests for DeepSeek V4 reasoning parsing with injected reasoning mode enabled, covering tool-continuation scenarios, </think> token boundary handling, and content/reasoning separation.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The pull request description is comprehensive and well-structured, covering all required template sections with detailed technical information.
Title check ✅ Passed The title accurately describes the core change: removing a special-case adjustment for prompt_injected_reasoning when dealing with tool messages, which directly addresses DeepSeek V4 tool-continuation and reasoning parsing behavior.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@biswapanda
biswapanda marked this pull request as draft April 30, 2026 20:14
@biswapanda
biswapanda force-pushed the bis/dsv4-tool-plus-think branch from 93bba90 to dcc5786 Compare May 1, 2026 20:59
@copy-pr-bot

copy-pr-bot Bot commented May 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@biswapanda
biswapanda force-pushed the bis/dsv4-tool-plus-think branch from dcc5786 to 3abb3ba Compare May 1, 2026 21:02
@biswapanda
biswapanda marked this pull request as ready for review May 1, 2026 21:07
@biswapanda
biswapanda enabled auto-merge (squash) May 1, 2026 21:10
@biswapanda
biswapanda disabled auto-merge May 1, 2026 23:45
@biswapanda
biswapanda force-pushed the bis/dsv4-tool-plus-think branch from 3abb3ba to e2d3487 Compare May 1, 2026 23:50
@biswapanda biswapanda changed the title fix: dsv4 tool and reasoning parser fix: deepseek v4 inter-leaved tool and reasoning parser May 1, 2026
@biswapanda

Copy link
Copy Markdown
Contributor Author

Verified against DSV4 deployment -

POST http://localhost:8000/v1/chat/completions model=deepseek-ai/DeepSeek-V4-Pro
Replaying: user -> assistant(tool_call=bash) -> tool(Hello, world!) -> ?
----- streaming -----
[reason] The
[reason] \ script
[reason] \ ran
[reason] \ successfully
[reason] \ and
[reason] \ printed
[reason] \ "
[reason] Hello
[reason] ,
[reason] \ world
[reason] !"
[reason] \ as
[reason] \ expected
[reason] .
[cont] Done
[cont] !
[cont] \ I
[cont] \ created
[cont] \ a
[cont] \ `
[cont] hello
[cont] .py
[cont] `
[cont] \ file
[cont] \ and
[cont] \ ran
[cont] \ it
[cont] \ —
[cont] \ it
[cont] \ printed
[cont] \ **
[cont] Hello
[cont] ,
[cont] \ world
[cont] !
[cont] **
[cont] \ successfully
[cont] .
[finish] stop
----- done -----

This branch had an error being deployed

1 failed deployment
GITLAB — e2d34877 Deployed May 2, 2026 by copy-pr-bot[bot] via Mirror Repository to GitLab #26015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants