use local llm train ,acc is 0 #28

Activity

1. qq594495953 commented on Jun 1, 2026

   

   qq594495953

   on Jun 1, 2026

   More actions

   我也是，不确定是不是API调用的问题
2. raytrun commented on Jun 1, 2026

   

   raytrun

   on Jun 1, 2026

   Contributor

   More actions

   Thanks for bringing this to our attention!

   An acc=0.000 across all rollouts typically suggests an infrastructure or format parsing mismatch rather than the model completely lacking the capability to solve the tasks.

   To help us narrow down the root cause, could you provide a bit more context?

   1. Information Needed

   • Your Training Config: Please share the YAML file passed to train.py.
   • Benchmark & Data: Which benchmark are you evaluating??
   • Raw Model Outputs: A snippet of the raw trajectory logs (under outputs/ or via --verbose) showing exactly what the target model generated for a few cn_item_* tasks.

   ---

   2. Common Pitfalls with Local/Qwen Models

   If you are running a local model deployment, this issue usually stems from one of the following:

   1. Prompt/Chat Template Mismatch: The evaluator might expect a strict format (e.g., JSON or specific delimiters). Local wrappers or chat templates sometimes append extra tokens or free-form reasoning text that breaks the parser.
   2. API Response Schema: If using an OpenAI-compatible endpoint (e.g., vLLM, Ollama, Xinference), ensure the returned JSON payload strictly adheres to the standard choices[0].message.content format expected by SkillOpt's backend.
3. lemon-ya-ha commented on Jun 2, 2026

   

   无敌小霸王 (lemon-ya-ha)

   on Jun 2, 2026

   Author

   More actions

   1. path: configs/searchqa/default.yaml
      default.yaml

   2. benchmark: searchqa
      data images show

   

   train.json

   test.json

   sel.json

   3.outputs

   skillopt_searchqa_Qwen-Qwen35-27B_20260602_085947.zip

   4. local llm config
      export QWEN_CHAT_BASE_URL=http://192.168.2.24:9999/v1/chat/completions
      export QWEN_CHAT_MODEL=Qwen/Qwen35-27B

   

   5. run train.py

   python scripts/train.py
   --config configs/searchqa/default.yaml
   --split_dir *******/SkillOpt/data/searchqa_text
   --qwen_chat_base_url http://192.168.2.24:9999/v1/chat/completions
   --optimizer_model Qwen/Qwen35-27B
   --target_model Qwen/Qwen35-27B
4. lemon-ya-ha commented on Jun 3, 2026

   

   无敌小霸王 (lemon-ya-ha)

   on Jun 3, 2026

   Author

   More actions

   Thanks for bringing this to our attention!

   An acc=0.000 across all rollouts typically suggests an infrastructure or format parsing mismatch rather than the model completely lacking the capability to solve the tasks.

   To help us narrow down the root cause, could you provide a bit more context?

   1. Information Needed

   • Your Training Config: Please share the YAML file passed to train.py.
   • Benchmark & Data: Which benchmark are you evaluating??
   • Raw Model Outputs: A snippet of the raw trajectory logs (under outputs/ or via --verbose) showing exactly what the target model generated for a few cn_item_* tasks.

   2. Common Pitfalls with Local/Qwen Models

   If you are running a local model deployment, this issue usually stems from one of the following:

   1. Prompt/Chat Template Mismatch: The evaluator might expect a strict format (e.g., JSON or specific delimiters). Local wrappers or chat templates sometimes append extra tokens or free-form reasoning text that breaks the parser.
   2. API Response Schema: If using an OpenAI-compatible endpoint (e.g., vLLM, Ollama, Xinference), ensure the returned JSON payload strictly adheres to the standard choices[0].message.content format expected by SkillOpt's backend.

   I have submitted the relevant attachments.
5. mvanhorn commented on Jun 7, 2026

   

   Matt Van Horn (mvanhorn)

   on Jun 7, 2026

   Contributor

   More actions

   Opened #40 for the acc=0 case on local vLLM targets: the router wasn't forwarding timeout to the Qwen backend, and enable_thinking was injected unconditionally (non-Qwen servers then emit <think> output with no <answer> tag). The PR forwards the timeout and only sets enable_thinking when enabled.

   👍React with 👍1Reacted by raytrun
6. Yifan Yang (Yif-Yang)

   closed this as completedin #40on Jun 7, 2026
7. pull

   added a commit that references this issue on Jun 7, 2026

   fix(model): forward Qwen timeout and only set enable_thinking when true

   ...

   This issue will close once commit c31c50b is merged into the 'main' branch.

   c31c50b
8. Yifan Yang (Yif-Yang)

   added a commit that references this issue on Aug 20, 2026

   Add explicit qwen_chat thinking_mode wire policy (#240)

   ...

   This issue will close once commit 601f5f7 is merged into the 'main' branch.

   601f5f7