Repository navigation
Clarification request: Qwen enable_thinking setting used for reported ALFWorld results #90
Description
Hi authors,
I have a question about the Qwen backend configuration used for the reported results, especially the ALFWorld numbers for Qwen3.6-35B-A3B or Qwen3.5–4B.
In the current released code, skillopt/model/qwen_backend.py appears to only send:
payload["chat_template_kwargs"] = {"enable_thinking": True}when config.enable_thinking is true. When enable_thinking=false, the request does not explicitly send chat_template_kwargs={"enable_thinking": False}. In that case, the actual thinking behavior may depend on the vLLM server / model chat-template default.
Could you clarify:
-
For the ALFWorld results reported in the paper for
Qwen3.6-35B-A3B, was Qwen thinking mode enabled or disabled? -
If it was disabled, did your serving endpoint explicitly force
enable_thinking=false, or was it controlled only by the SkillOpt client-side config? -
Would you recommend always sending
chat_template_kwargs={"enable_thinking": false}whenenable_thinking=falseto avoid ambiguity across vLLM deployments?
I am asking because local runs can differ substantially depending on whether Qwen reasoning/thinking is enabled by default on the serving side. Explicitly clarifying this would help make the reported Qwen results easier to reproduce.
Thanks!
Activity
yanan1116 commentedon Jul 2, 2026
raytrun could you take a look at this concern. thanks.
Huangzisu commentedon Jul 6, 2026
Hi Yanan (@yanan1116), sorry for the delay. Here is some clarification:
enable_thinking=false, that is, Qwen thinking mode is disabled in our reported experimental results.enable_thinking=falsefrom client side.Thanks for your question! Also, we plan to conduct relevant experiments with Qwen thinking mode enabled to provide more reference for users — stay tuned!
Huangzisu commentedon Jul 21, 2026
Hi Yanan (@yanan1116) , here are the corresponding results with Qwen thinking mode enabled:
Overall, the reasoning-enabled version generally performs better than the corresponding reasoning-disabled version. SkillOpt still works when we turn reasoning mode on. We do not tune training hyperparameters in these experiments. For better performance, we have multiple key training configs for user to explore. We may find a more suitable set of training hyperparameters for Qwen with reasoning mode enabled like we do when training neural networks.
Regarding your specific question about ALFWorld, enabling thinking mode does not consistently outperform the non-thinking setting. One possible reason is that the ALFWorld task prompt explicitly requires the model to generate a think tag in its output, which may partially conflict with or interfere with the model’s native reasoning behavior.
Add explicit qwen_chat thinking_mode wire policy (#240)