[Bugfix][Parser] Surface unterminated reasoning as content for deepseek_v4 (fixes #48645) - #50753

[Bugfix][Parser] Surface unterminated reasoning as content for deepseek_v4 (opt-in)

Problem

Fixes #48645 (the content-population half; the trailing-EOS half was
already fixed by #48748, merged 2026-07-22 — see "Scope" below).

With --reasoning-parser deepseek_v4 and thinking=true (the common
case: it's the server default via
--default-chat-template-kwargs={"thinking":true,...} on the
DeepSeek-V4-Flash recipe,
and applies even to OpenAI-compatible clients that never send
chat_template_kwargs), a generation that ends without ever
emitting </think> has its entire answer routed to
reasoning_content, leaving content empty:

{"role": "assistant",
 "reasoning_content": "Good morning! How can I help you today?",
 "content": ""}

Any caller that reads content only — most OpenAI-compatible chat
clients, and especially structured-output / tool-eval harnesses that
don't fall back to reasoning_content — sees an empty response even
though the model answered correctly.

This is sampling-dependent, not deterministic: issue reporter fank
measured ~20-25% misroute rates on a real agent system prompt (50 tool
defs, instructions to answer directly/without preamble) at
temperature 1.0, and 0% on generic/synthetic prompts of the same
length — the trigger is prompt content (bias toward answering
immediately), not length.

Real-world impact beyond the original report

• chatboxai/chatbox#3821
  (opened 2026-07-30, active through 2026-08-02): DeepSeek-V4 served via
  OpenRouter → StreamLake. Network-capture-diagnosed (client-side ruled
  out): delta.content empty, full answer in delta.reasoning,
  finish_reason: "stop" — exactly this misroute signature, hit in
  production chat traffic rather than a synthetic repro.
• NVIDIA DGX Spark forum, thread 372268,
  post #544 (2026-08-02):
  tool-eval-bench case TC-22 "Output Format Compliance" — a terse
  "respond with ONLY valid JSON, no other text" prompt (exactly the
  no-preamble short-reply trigger fank identified) — fails with "Did
  not return valid JSON as requested." immediately after operators
  enabled thinking_mode=thinking + high/max effort via the official
  chat template. The harness reads content only, consistent with the
  misroute returning "".

Root cause

DeepSeekV4Parser is built on the newer ParserEngine state-machine
architecture (vllm/parser/engine/). With thinking=true it starts
in initial_state=ParserState.REASONING
(vllm/parser/deepseek_v4.py), and the only way out is the
THINK_END (</think>) transition. Per maintainer bbrowning's
analysis
(2026-07-15), starting in REASONING is correct here — DeepSeek-V4's
prompt is constructed by custom Python logic (not a chat template) and
does open an unterminated <think> — so option 2 from the issue
("only enter REASONING on evidence of thinking") is not applicable to
this model. The bug is that the model doesn't always honor that
opening: on short replies it sometimes answers directly and hits EOS
without ever closing </think>. When that happens, every chunk stays
tagged REASONING_CHUNK for the whole response and content never
gets populated. This is a model behavior, not unique to DeepSeek-V4 —
bbrowning notes "I've only seen this happen with Nemotron models in
very short model responses, and I suspect it's a similar situation
here."

Fix

Ports the existing Nemotron V3 fallback (PR #39091,
vllm/parser/nemotron_v3.py) to vllm/parser/deepseek_v4.py, per
bbrowning's explicit recommendation on #48645: "We already have a
precedent for something like this in the Nemotron v3 model, where
there's a chat template kwarg flag that can force the model to always
output content by copying any reasoning into the content if we see
generation ended during reasoning. The parser is designed to allow
models to opt-in to some fallback behavior here." Adapted for
upstream main's ParserEngine architecture (Nemotron's own port
predates today's engine and required no changes beyond a signature
addition; DeepSeek-V4's did not exist before this PR).

• Opt-in, matching Nemotron's precedent exactly: enabled via
  chat_template_kwargs={"force_nonempty_content": True}. Default
  behavior (flag absent/false) is unchanged — no silent behavior
  change for existing deployments. (bbrowning's comment frames this as
  vendor/deployment preference, the same framing used for Nemotron; if
  maintainers would prefer default-on for deepseek_v4 specifically,
  that's a one-line change in _should_force_content — flagged as an
  open design question below.)
• Gated on finish_reason == "stop" only (streaming path): never
  flushes on length-truncation. A response cut off by max_tokens
  still has an unterminated reasoning block, but that's truncated CoT,
  not a "the model skipped thinking" case — it must never become
  user-facing content. This required threading finish_reason through
  Parser.parse_delta → DelegatingParser.finalize_generation →
  ParserEngine.get_streaming_fallback_content, and both
  serving.py call sites (chat_completion and responses), since
  finish_reason wasn't previously available at the fallback-decision
  point for any parser, including Nemotron's.
• Never flushes when </think> was actually seen. The state
  machine's own .finish() emits a synthetic REASONING_END for
  any input still in REASONING at end-of-stream, so a naive
  "check reasoning_ended after calling the base implementation"
  is always true and can't distinguish a real close from EOF-in-
  REASONING. extract_reasoning() instead captures whether a real
  REASONING_END event arrived from _feed() before merging in
  .finish()'s synthetic one.
• Never flushes when any tool call was emitted in the same turn
  (checked via self._tool_slots).

Scope: not re-fixing the EOS half

Upstream PR #48748 (merged 2026-07-22) already fixed trailing-EOS
leakage into reasoning_content for deepseek_v4 — its own test,
test_eos_not_leaked_when_reasoning_never_ends (pre-existing, kept
as-is, still passing), asserts the EOS text does not appear anywhere,
while also asserting output.content == "". That test's own
assertion is proof that #48748 explicitly left the core misroute (the
subject of this PR) unfixed — it only cleaned up the special token,
not the routing. This PR builds on current main (which includes
#48748) and does not touch EOS-stripping logic; it only adds the
opt-in content-population fallback on top.

What upstream's architecture forced differently from a literal Nemotron port

• extract_reasoning() does not call super(). It reimplements
  ParserEngine.extract_reasoning's body, because the synthetic-
  REASONING_END issue above means the natural-vs-synthetic
  distinction has to be captured before the parent method's own
  internal call to .finish().
• Non-streaming has no finish_reason gate. extract_reasoning()
  is called with only model_output: str — no finish_reason is
  available at that call site network-wide (touching that would mean
  changing the shared extract_reasoning signature used by ~15
  unrelated legacy reasoning parsers, out of scope here). Non-streaming
  therefore only gates on natural-end/tool-slots, not truncation. This
  is a real, intentional gap, called out explicitly in
  test_non_streaming_has_no_finish_reason_gate, which documents that
  a truncated/never-closed non-streaming trace does get swapped when
  the opt-in flag is set. Flagged as an open design question below.
• finish_reason plumbing is broader than Nemotron needed. Since
  Nemotron's own fallback predates this PR and has no truncation gate,
  adding one for deepseek_v4 meant extending the shared
  Parser.parse_delta / DelegatingParser.finalize_generation /
  ParserEngine.parse_delta / ParserEngineReasoningAdapter surface
  (plus the two serving.py call sites) with an additive, defaulted
  finish_reason: str | None = None keyword — touching 4 parser
  implementations' parse_delta signatures for interface completeness
  (harmony.py, kimi_k3.py unaffected in behavior) and left
  Nemotron's own fallback ungated on finish_reason (unchanged
  behavior, documented with a comment) since changing an existing,
  shipped default was out of scope for this PR.

Test coverage

Mirrors tests/parser/engine/test_nemotron_v3.py. Added to
tests/parser/engine/test_deepseek_v4.py:

Non-streaming (TestForceNonemptyContentNonStreaming, 5 tests):
misroute-flushed, no-flush-without-opt-in, end-token-seen-not-flushed,
tool-call-not-flushed, and the documented finish_reason-gap test.

Streaming (TestForceNonemptyContentStreaming, 7 tests, most
parametrized over chunk sizes 1/2/3/5/None for chunk-boundary
invariance): misroute-flushed, no-flush-without-opt-in,
length-truncation-not-flushed (finish_reason="length"),
no-finish-reason-not-flushed (deny-by-default when finish_reason is
unset), end-token-seen-not-flushed, tool-call-not-flushed (tokens
built with real DSML vocab IDs so the token-ID-confirmed terminal
matcher fires correctly), and flushed-content-has-no-trailing-eos
(proves flushed content inherits #48748's EOS-clean guarantee).

Full suite: tests/parser/engine/test_deepseek_v4.py 90/90 passed
(32 new). Full tests/parser/ suite: 3768 passed (32 new), 19 failed
/ 285 errors — failure/error ID sets byte-identical to a pre-change
baseline run, i.e. zero regressions. (The pre-existing failures/errors
are environment-related — e.g. missing optional/compiled deps in this
sandboxed check — not related to this change; confirmed unchanged
before vs. after.)

ruff check / ruff format: clean on every file this PR touches.
(One pre-existing ISC004 finding at
tests/parser/engine/test_deepseek_v4.py:220 predates this PR —
verified via git diff that this PR's changes are purely additive
starting after the file's existing content — and is left untouched.)

Production validation

Deployed on 2x ASUS GX10 (GB10), tensor-parallel=2, serving
DeepSeek-V4-Flash-0731. The deterministic parser-level repro from the
issue (reasoning_effort: "none" with server-default thinking:true)
is fixed. Across finish=length truncation probes, closed-</think>-
then-stop turns, tool-call turns, and default max-effort traffic:
zero false flushes — flush events were log-counted and matched
expectations exactly (fires only on the intended misroute case, never
on truncation, a real close, or a tool-call turn).

References

• Fixes [Bug]: deepseek_v4 parser: reply without </think> routes the whole answer to reasoning_content (content empty), trailing EOS not stripped #48645
• Ports [Bugfix][Reasoning] Nemotron V3: surface reasoning as content when thinking is unterminated #39091 (Nemotron V3 force_nonempty_content precedent)
• Builds on [Bugfix][Parser] Fix special tokens (EOS/BOS) leaking into reasoning content #48748 (EOS-stripping, already merged; not duplicated here)
• bbrowning's guidance: [Bug]: deepseek_v4 parser: reply without </think> routes the whole answer to reasoning_content (content empty), trailing EOS not stripped #48645 (comment)

Open design questions for maintainers

1. Opt-in vs. default-on for deepseek_v4. bbrowning's comment
   frames the Nemotron precedent as vendor-preference opt-in, and this
   PR follows that exactly. But [Bug]: deepseek_v4 parser: reply without </think> routes the whole answer to reasoning_content (content empty), trailing EOS not stripped #48645's own reporter and both
   real-world reports above show this hurts out-of-the-box deployments
   using the documented recipe's default thinking:true — none of
   those callers know to set force_nonempty_content. Happy to flip
   the default for deepseek_v4 specifically (one line in
   _should_force_content) if maintainers prefer that over
   Nemotron-style opt-in.
2. Non-streaming finish_reason gap (see above) — accept as
   documented, or worth threading finish_reason through the shared
   extract_reasoning signature in a follow-up (broader blast radius,
   touches every reasoning parser)?
3. Nemotron's fallback stays ungated on finish_reason even
   though the new gate exists on the shared path now — intentionally
   left unchanged to avoid altering shipped Nemotron behavior in this
   PR. Separate follow-up if maintainers want parity.