Skip to content

Commit 601f5f7

Browse files
authored
Add explicit qwen_chat thinking_mode wire policy (#240)
chat_template_kwargs is a vLLM/SGLang extension. OpenAI, Azure, and strict OpenAI-compatible gateways reject the unknown body field with HTTP 400, and non-Qwen vLLM models served with it can emit <think> output with no <answer> tag (acc=0.000). c31c50b fixed that by only emitting the field when thinking was enabled, which closed #28 but left no supported way to send an explicit enable_thinking: false -- the request in #90/#109. The protocol has three states, so make the setting three-state: server_default (default) -> omit chat_template_kwargs enabled -> send enable_thinking: true disabled -> send enable_thinking: false server_default keeps every existing deployment on exactly the bytes it sends today, so #28 stays fixed, while disabled gives #90 the explicit false it asks for. The legacy enable_thinking boolean keeps its historical wire meaning (true -> send true, false -> omit), so no config changes behavior; setting both keys to conflicting values raises rather than silently picking a winner. Unknown tokens raise too -- a typo must not silently flip a reproducibility control. Because server_default delegates a result-affecting choice to the server's chat template, the backend warns once per role when it is used, and the resolved per-role mode is recorded in the run's config.json under resolved_qwen_thinking_modes. Also settles the docs contradiction between "local vLLM endpoint" and "OpenAI-compatible": qwen_chat speaks the OpenAI protocol and reaches both self-hosted servers and hosted gateways, which is exactly why the wire policy cannot be inferred and must be explicit. Closes #90
1 parent a11f377 commit 601f5f7

11 files changed

Lines changed: 2848 additions & 2477 deletions

File tree

‎.env.example‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,13 @@ export AZURE_OPENAI_AUTH_MODE=api_key
5050
# does not create a direct Anthropic API client for this backend.
5151
# export ANTHROPIC_API_KEY=sk-ant-...
5252

53-
# ── Qwen Local Model (for qwen_chat backend) ────────────────────────
53+
# ── Qwen via an OpenAI-compatible endpoint (qwen_chat backend) ──────
5454
# export QWEN_CHAT_BASE_URL=http://localhost:8000/v1
5555
# export QWEN_CHAT_MODEL=Qwen/Qwen3.5-4B
56+
# Wire policy for chat_template_kwargs.enable_thinking, a vLLM/SGLang
57+
# extension that OpenAI/Azure and strict gateways reject with HTTP 400:
58+
# server_default (default) omit the field | enabled | disabled
59+
# export QWEN_CHAT_THINKING_MODE=disabled
5660
# The train/eval entry points likewise override this model with
5761
# model.optimizer/model.target for the selected Qwen roles.
5862

‎configs/_base_/default.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,14 @@ model:
5454
target_azure_openai_ad_scope: "https://cognitiveservices.azure.com/.default"
5555
target_azure_openai_managed_identity_client_id: ""
5656

57+
# Qwen backend settings (qwen_chat roles). `chat_template_kwargs` is a
58+
# vLLM/SGLang extension that OpenAI/Azure reject, so the wire policy is
59+
# explicit: server_default omits the key (safe default), enabled/disabled
60+
# send it. Pin "disabled" to reproduce the published non-thinking numbers.
61+
qwen_chat_thinking_mode: "" # server_default (default) | enabled | disabled
62+
optimizer_qwen_chat_thinking_mode: "" # per-role override
63+
target_qwen_chat_thinking_mode: "" # per-role override
64+
5765
# MiniMax backend settings (minimax_chat target)
5866
minimax_region: "" # global_en (default) or cn_zh; selects the base URL
5967
minimax_base_url: "" # region base URL if blank

‎docs/guide/configuration.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ model:
4141
| `openai_chat` | ✓ | ✓ | Azure OpenAI, or its explicit compatibility auth mode |
4242
| `openai_compatible` | ✓ | ✓ | Generic OpenAI Chat Completions endpoint |
4343
| `claude_chat` | ✓ | ✓ | Claude Code CLI (`claude -p`) |
44-
| `qwen_chat` | ✓ | ✓ | Qwen served through an OpenAI-compatible local endpoint |
44+
| `qwen_chat` | ✓ | ✓ | Qwen served through an OpenAI-compatible endpoint (self-hosted vLLM/SGLang or a hosted gateway) |
4545
| `minimax_chat` | ✓ | ✓ | MiniMax API |
4646
| `copilot_chat` | ✓ | ✓ | GitHub Copilot CLI (`copilot -p`); alias `copilot` |
4747
| `codex_exec` | — | ✓ | Codex CLI execution harness |
@@ -204,7 +204,8 @@ Model credentials are loaded from environment variables:
204204
| `CURSOR_EXEC_PATH` | `cursor_exec` | Optional path to `cursor-agent`; defaults to `cursor-agent` |
205205
| `CURSOR_EXEC_SANDBOX` | `cursor_exec` | Cursor sandbox mode: `enabled` (default) or `disabled` |
206206
| `CURSOR_API_KEY` | `cursor_exec` | Optional authentication method understood directly by Cursor Agent |
207-
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | Local Qwen/vLLM endpoint |
207+
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | OpenAI-compatible Qwen endpoint: self-hosted vLLM/SGLang or a hosted gateway |
208+
| `QWEN_CHAT_THINKING_MODE` | `qwen_chat` | `server_default` (default; omit `chat_template_kwargs`), `enabled`, or `disabled`; per-role `OPTIMIZER_`/`TARGET_` variants take precedence |
208209
| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence |
209210
| `MINIMAX_REGION` | `minimax_chat` | Service region: `global_en` (default) or `cn_zh`; selects the base URL |
210211
| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL; overrides the region default |

‎docs/reference/config.md‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,12 +61,42 @@ Every shared key also has an `optimizer_azure_openai_*` and
6161
defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`.
6262
`ANTHROPIC_API_KEY` is one authentication option understood by the CLI.
6363

64+
### Qwen thinking mode
65+
66+
`qwen_chat` speaks the OpenAI chat-completions protocol, so it reaches both
67+
self-hosted servers (vLLM, SGLang) and hosted OpenAI-compatible gateways.
68+
`chat_template_kwargs` is a vLLM/SGLang extension: OpenAI, Azure OpenAI, and
69+
strict gateways reject the unknown body field with HTTP 400, and non-Qwen vLLM
70+
models served with it may emit `<think>` output without an `<answer>` tag.
71+
Because the correct wire policy therefore depends on the serving stack rather
72+
than on a boolean preference, it is an explicit three-state setting:
73+
74+
| `thinking_mode` | On the wire |
75+
|---|---|
76+
| `server_default` (default) | `chat_template_kwargs` is **not sent**; the server's chat template decides |
77+
| `enabled` | sends `chat_template_kwargs: {"enable_thinking": true}` |
78+
| `disabled` | sends `chat_template_kwargs: {"enable_thinking": false}` |
79+
80+
`server_default` is the portable default and works against any
81+
OpenAI-compatible endpoint, but Qwen3 chat templates enable thinking by
82+
default, so the outcome depends on the serving stack and template version. The
83+
backend warns once per role when a request is sent under `server_default`. For
84+
reproducible runs pin `enabled` or `disabled`; the resolved per-role mode is
85+
recorded in the run's `config.json` under `resolved_qwen_thinking_modes`.
86+
87+
The legacy `model.qwen_chat_enable_thinking` boolean keeps its historical
88+
meaning exactly — `true` sends `enable_thinking: true`, `false` omits the field
89+
(it never sent an explicit `false`) — so existing configs are unaffected.
90+
Setting both keys to conflicting values raises an error rather than picking a
91+
winner. Prefer `thinking_mode`; use `disabled` when you need the field sent.
92+
6493
### Qwen, MiniMax, and Exec Backends
6594

6695
| Parameter family | Description |
6796
|---|---|
68-
| `model.qwen_chat_*` | Shared `base_url`, `api_key`, `temperature`, `timeout_seconds`, `max_tokens`, and `enable_thinking` |
69-
| `model.optimizer_qwen_chat_*` / `model.target_qwen_chat_*` | Per-role Qwen overrides |
97+
| `model.qwen_chat_*` | Shared `base_url`, `api_key`, `temperature`, `timeout_seconds`, `max_tokens`, `thinking_mode`, and the legacy `enable_thinking` |
98+
| `model.qwen_chat_thinking_mode` | Wire policy for `chat_template_kwargs.enable_thinking`: `server_default` (default; omit the field), `enabled`, or `disabled`. See [Qwen thinking mode](#qwen-thinking-mode) |
99+
| `model.optimizer_qwen_chat_*` / `model.target_qwen_chat_*` | Per-role Qwen overrides, including `*_qwen_chat_thinking_mode` |
70100
| `model.minimax_*` | MiniMax `region`, `base_url`, `api_key`, shared `minimax_model`, `temperature`, `max_tokens`, and `enable_thinking`; `minimax_model` applies when MiniMax is the target |
71101
| `model.codex_exec_*` | Codex path, sandbox, profile, SDK mode, reasoning, network/search, and approval policy; see compatibility notes below |
72102
| `model.claude_code_exec_*` | Claude path, profile, SDK mode, effort, and thinking-token cap |

‎scripts/eval_only.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -525,12 +525,14 @@ def _set_role(key: str, value: str) -> None:
525525
timeout_seconds=cfg.get("qwen_chat_timeout_seconds"),
526526
max_tokens=cfg.get("qwen_chat_max_tokens"),
527527
enable_thinking=cfg.get("qwen_chat_enable_thinking"),
528+
thinking_mode=cfg.get("qwen_chat_thinking_mode"),
528529
target_base_url=cfg.get("target_qwen_chat_base_url") or None,
529530
target_api_key=cfg.get("target_qwen_chat_api_key") or None,
530531
target_temperature=cfg.get("target_qwen_chat_temperature"),
531532
target_timeout_seconds=cfg.get("target_qwen_chat_timeout_seconds"),
532533
target_max_tokens=cfg.get("target_qwen_chat_max_tokens"),
533534
target_enable_thinking=cfg.get("target_qwen_chat_enable_thinking"),
535+
target_thinking_mode=cfg.get("target_qwen_chat_thinking_mode"),
534536
)
535537
configure_minimax_chat(
536538
region=cfg.get("minimax_region") or None,

‎scripts/train.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,18 +180,21 @@ def parse_args() -> argparse.Namespace:
180180
p.add_argument("--qwen_chat_timeout_seconds", type=float)
181181
p.add_argument("--qwen_chat_max_tokens", type=int)
182182
p.add_argument("--qwen_chat_enable_thinking", type=_BOOL)
183+
p.add_argument("--qwen_chat_thinking_mode", type=str)
183184
p.add_argument("--optimizer_qwen_chat_base_url", type=str)
184185
p.add_argument("--optimizer_qwen_chat_api_key", type=str)
185186
p.add_argument("--optimizer_qwen_chat_temperature", type=float)
186187
p.add_argument("--optimizer_qwen_chat_timeout_seconds", type=float)
187188
p.add_argument("--optimizer_qwen_chat_max_tokens", type=int)
188189
p.add_argument("--optimizer_qwen_chat_enable_thinking", type=_BOOL)
190+
p.add_argument("--optimizer_qwen_chat_thinking_mode", type=str)
189191
p.add_argument("--target_qwen_chat_base_url", type=str)
190192
p.add_argument("--target_qwen_chat_api_key", type=str)
191193
p.add_argument("--target_qwen_chat_temperature", type=float)
192194
p.add_argument("--target_qwen_chat_timeout_seconds", type=float)
193195
p.add_argument("--target_qwen_chat_max_tokens", type=int)
194196
p.add_argument("--target_qwen_chat_enable_thinking", type=_BOOL)
197+
p.add_argument("--target_qwen_chat_thinking_mode", type=str)
195198
p.add_argument("--minimax_region", type=str)
196199
p.add_argument("--minimax_base_url", type=str)
197200
p.add_argument("--minimax_api_key", type=str)
@@ -401,18 +404,21 @@ def _retired_option_sources(
401404
"qwen_chat_timeout_seconds": "model.qwen_chat_timeout_seconds",
402405
"qwen_chat_max_tokens": "model.qwen_chat_max_tokens",
403406
"qwen_chat_enable_thinking": "model.qwen_chat_enable_thinking",
407+
"qwen_chat_thinking_mode": "model.qwen_chat_thinking_mode",
404408
"optimizer_qwen_chat_base_url": "model.optimizer_qwen_chat_base_url",
405409
"optimizer_qwen_chat_api_key": "model.optimizer_qwen_chat_api_key",
406410
"optimizer_qwen_chat_temperature": "model.optimizer_qwen_chat_temperature",
407411
"optimizer_qwen_chat_timeout_seconds": "model.optimizer_qwen_chat_timeout_seconds",
408412
"optimizer_qwen_chat_max_tokens": "model.optimizer_qwen_chat_max_tokens",
409413
"optimizer_qwen_chat_enable_thinking": "model.optimizer_qwen_chat_enable_thinking",
414+
"optimizer_qwen_chat_thinking_mode": "model.optimizer_qwen_chat_thinking_mode",
410415
"target_qwen_chat_base_url": "model.target_qwen_chat_base_url",
411416
"target_qwen_chat_api_key": "model.target_qwen_chat_api_key",
412417
"target_qwen_chat_temperature": "model.target_qwen_chat_temperature",
413418
"target_qwen_chat_timeout_seconds": "model.target_qwen_chat_timeout_seconds",
414419
"target_qwen_chat_max_tokens": "model.target_qwen_chat_max_tokens",
415420
"target_qwen_chat_enable_thinking": "model.target_qwen_chat_enable_thinking",
421+
"target_qwen_chat_thinking_mode": "model.target_qwen_chat_thinking_mode",
416422
"minimax_region": "model.minimax_region",
417423
"minimax_base_url": "model.minimax_base_url",
418424
"minimax_api_key": "model.minimax_api_key",

‎skillopt/config.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,18 +100,21 @@
100100
"model.qwen_chat_timeout_seconds": "qwen_chat_timeout_seconds",
101101
"model.qwen_chat_max_tokens": "qwen_chat_max_tokens",
102102
"model.qwen_chat_enable_thinking": "qwen_chat_enable_thinking",
103+
"model.qwen_chat_thinking_mode": "qwen_chat_thinking_mode",
103104
"model.optimizer_qwen_chat_base_url": "optimizer_qwen_chat_base_url",
104105
"model.optimizer_qwen_chat_api_key": "optimizer_qwen_chat_api_key",
105106
"model.optimizer_qwen_chat_temperature": "optimizer_qwen_chat_temperature",
106107
"model.optimizer_qwen_chat_timeout_seconds": "optimizer_qwen_chat_timeout_seconds",
107108
"model.optimizer_qwen_chat_max_tokens": "optimizer_qwen_chat_max_tokens",
108109
"model.optimizer_qwen_chat_enable_thinking": "optimizer_qwen_chat_enable_thinking",
110+
"model.optimizer_qwen_chat_thinking_mode": "optimizer_qwen_chat_thinking_mode",
109111
"model.target_qwen_chat_base_url": "target_qwen_chat_base_url",
110112
"model.target_qwen_chat_api_key": "target_qwen_chat_api_key",
111113
"model.target_qwen_chat_temperature": "target_qwen_chat_temperature",
112114
"model.target_qwen_chat_timeout_seconds": "target_qwen_chat_timeout_seconds",
113115
"model.target_qwen_chat_max_tokens": "target_qwen_chat_max_tokens",
114116
"model.target_qwen_chat_enable_thinking": "target_qwen_chat_enable_thinking",
117+
"model.target_qwen_chat_thinking_mode": "target_qwen_chat_thinking_mode",
115118
"model.minimax_region": "minimax_region",
116119
"model.minimax_base_url": "minimax_base_url",
117120
"model.minimax_api_key": "minimax_api_key",

0 commit comments

Comments
 (0)