Skip to content

fix(models): forward reasoning_content as top-level field in multi-turn#1396

Merged
Yunnglin merged 2 commits into
mainfrom
fix/reasoning-history-as-field
Jun 4, 2026
Merged

fix(models): forward reasoning_content as top-level field in multi-turn#1396
Yunnglin merged 2 commits into
mainfrom
fix/reasoning-history-as-field

Conversation

@Yunnglin

@Yunnglin Yunnglin commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #1392 — DeepSeek V4 thinking mode rejects evalscope's multi-turn requests with HTTP 400 "The reasoning_content in the thinking mode must be passed back to the API.", and Qwen3 thinking silently pollutes context by parsing <think>...</think> as plain text.

Root cause: openai_assistant_content encodes ContentReasoning as <think>...</think> inside the content string and never writes a top-level reasoning_content field. This worked for OpenAI/Together/Groq but breaks modern OpenAI-compatible providers that expect reasoning_content as an independent field.

Fix: redefine GenerateConfig.reasoning_history literal from inspect_ai-style filter values ('none'|'all'|'last'|'auto') to wire-format values ('none'|'think_tag'|'reasoning_field'), and consume it in OpenAICompatibleAPI.generate / generate_async to control the per-request wire shape.

⚠ Breaking Change

Provider Old default (think_tag) New default (reasoning_field)
OpenAI / vLLM / SGLang / Together / Groq ✓ (extra field ignored)
DeepSeek V4 thinking ✗ 400 auto-fixed
Qwen3 thinking (DashScope) ⚠ context pollution auto-fixed
DeepSeek R1 (legacy deepseek-reasoner) 400 — opt-out required

R1 users must explicitly set --generation-config '{\"reasoning_history\":\"none\"}' because R1's protocol forbids reasoning_content echo-back. Documented in docs/{zh,en}/get_started/parameters.md.

Affected benchmarks before the fix

Drive-by cleanups in GenerateConfig

  • Remove the cargo-culted migrate_reasoning bool→literal validator (zero consumers since v1.0)
  • Remove 4 zombie fields with zero repo-wide consumers: best_of, cache_prompt, internal_tools, max_tool_output
  • Document batch_size as an internal field (synced from TaskConfig.eval_batch_size, not user-facing)
  • Document 6 previously-undocumented but actively-consumed fields: stop_seqs, repetition_penalty, response_schema, reasoning_effort, reasoning_tokens, reasoning_summary. Fix timeout type (intint/float)

Design notes

  • Last ContentReasoning wins when multiple are present on an assistant message (matches litellm's behavior; see _extract_last_reasoning).
  • Blank/whitespace reasoning falls back to a single-space \" \" placeholder with a debug log (DeepSeek V4 rejects empty reasoning_content).
  • No capability flag / model-name sniffing; users opt in/out explicitly via reasoning_history.
  • litellm_compatible.py path is intentionally out of scope (litellm has its own schema validation that may strip extra top-level keys — deferred to a follow-up PR).

Prior art: BerriAI/litellm#28080 — the same fix for the DeepSeek path in litellm, including the single-space fallback design.

Test plan

  • make lint passes
  • pytest tests/models/test_openai_reasoning.py -v — 12 new unit tests covering 3 modes × 5 message shapes + schema invariants (legacy literal rejection, new literal acceptance, default behavior)
  • pytest tests/agent/external/test_openai_chat_round_trip.py -v — 6 round-trip tests still green (regression check)
  • pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings — CI smoke green (no regression on the happy-path benchmark run)
  • End-to-end against qwen3-max (DashScope, thinking enabled): all 3 modes produce the expected wire shape (<think> in content / top-level reasoning_content / clean drop) and all 3 are accepted by the server
  • Reviewer: verify against DeepSeek V4 (deepseek-v4-pro + extra_body={\"thinking\":{\"type\":\"enabled\"}}) returns 200 instead of 400
  • Reviewer: verify DeepSeek R1 (deepseek-reasoner) regression — default config should now 400; reasoning_history='none' should return 200

…rn requests

Redefine GenerateConfig.reasoning_history from inspect_ai-style filter
values ('none'|'all'|'last'|'auto') to evalscope-style wire-format values
('none'|'think_tag'|'reasoning_field'), and consume it in
OpenAICompatibleAPI so the encoded shape of prior-turn reasoning_content
can be controlled per-request.

BREAKING CHANGE: default wire format switches from `<think>...</think>`
inline encoding to a top-level `reasoning_content` field, matching the
modern OpenAI-compatible reasoning protocol. Fixes DeepSeek V4 thinking
400 errors and removes context pollution for Qwen3 thinking. DeepSeek R1
(legacy) users must now explicitly opt out via `reasoning_history='none'`.

Also clean up GenerateConfig:
- Remove the cargo-culted `migrate_reasoning` bool->literal validator
  (zero consumers since v1.0).
- Remove 4 zombie fields with no consumers anywhere in the repo:
  best_of, cache_prompt, internal_tools, max_tool_output.
- Document `batch_size` as an internal field synced from
  TaskConfig.eval_batch_size (not user-facing).
- Document 6 previously-undocumented but actively-consumed fields:
  stop_seqs, repetition_penalty, response_schema, reasoning_effort,
  reasoning_tokens, reasoning_summary. Fix `timeout` type (int -> int/float).

Refs: #1392
Prior art: BerriAI/litellm#28080
Copilot AI review requested due to automatic review settings June 4, 2026 05:59

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the generation configuration and OpenAI-compatible model handling to support modern reasoning models (such as DeepSeek V4 and Qwen3 thinking modes). It replaces the legacy reasoning_history options with a new reasoning_format parameter supporting think_tag, reasoning_field, and none modes. This allows proper encoding of reasoning content either inline, as a top-level reasoning_content field, or stripping it entirely. The documentation and unit tests have been updated accordingly. The reviewer suggested ensuring backward compatibility in openai_assistant_content by allowing reasoning_format to accept a boolean value (the legacy include_reasoning parameter) and normalizing it internally.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread evalscope/models/utils/openai.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes multi-turn OpenAI-compatible request serialization for models/providers that require prior-turn reasoning_content to be echoed back as a top-level field (not embedded inside content as <think>...</think>). It introduces an explicit reasoning_history wire-format mode and updates the OpenAI-compatible API path to emit reasoning_content when configured (defaulting to the new reasoning_field behavior at consumption time).

Changes:

  • Add ReasoningFormat support to OpenAI chat message serialization, including promoting the last ContentReasoning to top-level reasoning_content and a blank-reasoning single-space fallback for strict servers.
  • Redefine GenerateConfig.reasoning_history literal values to match wire formats (none | think_tag | reasoning_field) and consume it in OpenAICompatibleAPI.generate / generate_async (defaulting to reasoning_field when unset).
  • Add unit tests for the three reasoning wire shapes and update parameter docs (EN/ZH) to document the new behavior and options.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/models/test_openai_reasoning.py Adds unit coverage pinning the outgoing assistant message wire shape across think_tag, reasoning_field, and none, plus schema validation for reasoning_history.
evalscope/models/utils/openai.py Adds reasoning_format to OpenAI chat message serialization, promotes reasoning_content as a top-level field in reasoning_field mode, and introduces _extract_last_reasoning.
evalscope/models/openai_compatible.py Passes reasoning_history through to openai_chat_messages, defaulting to reasoning_field when unset to fix strict providers (e.g., DeepSeek V4 thinking).
evalscope/api/model/generate_config.py Updates reasoning_history allowed literals/documentation and removes a legacy migration validator and unused fields from the typed schema.
docs/zh/get_started/parameters.md Documents reasoning_history modes and newly documented generation config fields; updates timeout type to int/float.
docs/en/get_started/parameters.md Same as ZH: documents reasoning_history modes/behavior and updates timeout type to int/float.

@Yunnglin Yunnglin merged commit a678e2d into main Jun 4, 2026
3 checks passed
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
…rn (modelscope#1396)

* fix(models): forward reasoning_content as top-level field in multi-turn requests

Redefine GenerateConfig.reasoning_history from inspect_ai-style filter
values ('none'|'all'|'last'|'auto') to evalscope-style wire-format values
('none'|'think_tag'|'reasoning_field'), and consume it in
OpenAICompatibleAPI so the encoded shape of prior-turn reasoning_content
can be controlled per-request.

BREAKING CHANGE: default wire format switches from `<think>...</think>`
inline encoding to a top-level `reasoning_content` field, matching the
modern OpenAI-compatible reasoning protocol. Fixes DeepSeek V4 thinking
400 errors and removes context pollution for Qwen3 thinking. DeepSeek R1
(legacy) users must now explicitly opt out via `reasoning_history='none'`.

Also clean up GenerateConfig:
- Remove the cargo-culted `migrate_reasoning` bool->literal validator
  (zero consumers since v1.0).
- Remove 4 zombie fields with no consumers anywhere in the repo:
  best_of, cache_prompt, internal_tools, max_tool_output.
- Document `batch_size` as an internal field synced from
  TaskConfig.eval_batch_size (not user-facing).
- Document 6 previously-undocumented but actively-consumed fields:
  stop_seqs, repetition_penalty, response_schema, reasoning_effort,
  reasoning_tokens, reasoning_summary. Fix `timeout` type (int -> int/float).

Refs: modelscope#1392
Prior art: BerriAI/litellm#28080

* chore: fix isort import grouping for new test file
@Yunnglin Yunnglin deleted the fix/reasoning-history-as-field branch June 10, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DeepSeek模型多轮对话时reasoning_content字 段处理不当导致评测失败

2 participants