Skip to content

vLLM 0.19 may lose tool calls for Qwen/Qwen3.5-35B-A3B-FP8 when XML tool_call is emitted inside <think> #39056

Open

Description

@ZenoAFfectionate

Problem

With vLLM 0.19, when serving the model:

  • Qwen/Qwen3.5-35B-A3B-FP8

using:

  • --reasoning-parser qwen3
  • --tool-call-parser qwen3_coder

non-streaming tool-call parsing can fail if the model emits XML tool-call markup inside the reasoning region, for example:

<think>
...
<tool_call>
<function=Finish>
<parameter=answer>
204
</parameter>
</function>
</tool_call>
</think>

In this case, the model may have produced a valid tool call, but the OpenAI-compatible response ends up with populated reasoning and empty tool_calls.

Observed model

The issue was observed and reproduced with:

  • Qwen/Qwen3.5-35B-A3B-FP8

It may also affect other Qwen3 / Qwen3.5 models that use the same parser combination, but the confirmed reproduction here is specifically on Qwen/Qwen3.5-35B-A3B-FP8.

Why this happens

The issue appears to come from the interaction between the reasoning parser and the tool parser:

  1. qwen3_reasoning_parser extracts everything before </think> into reasoning.
  2. downstream tool parsing only inspects content.
  3. if <tool_call>...</tool_call> remains inside reasoning, it never reaches qwen3_coder.

So the bug is not that vLLM makes Qwen3.5 generate tool calls inside <think>.
The bug is that vLLM currently does not recover those tool calls when that output pattern occurs.

Scope

This report is specifically about the non-streaming path, which can break benchmark / agent flows that rely on structured tool_calls being returned.

Proposed fix

I have opened a PR with a minimal parser-side fix here:

That patch promotes embedded XML tool-call blocks out of reasoning into content in qwen3_reasoning_parser, so the existing qwen3_coder tool parser can still parse them.

Request

Please consider reviewing and merging the PR if the approach looks acceptable.

Activity

  1. changed the title [-]vLLM 0.19 may lose Qwen3.5 tool calls when XML tool_call is emitted inside <think>[/-] [+]vLLM 0.19 may lose tool calls for Qwen/Qwen3.5-35B-A3B-FP8 when XML tool_call is emitted inside <think>[/+] on Apr 6, 2026
  2. hav4ik commented on Apr 6, 2026

    @hav4ik

    I've been observing similar effect with nvidia/Nemotron-Cascade2-30B-A3B as well (it uses qwen3_coder tool parser).

  3. Mr-Righter commented on Apr 6, 2026

    @Mr-Righter
    Image

    Deploying Qwen/Qwen3.5-122B-A10B-FP8 with the following arguments:

    • --model Qwen/Qwen3.5-122B-A10B-FP8
    • --async-scheduling
    • --tensor-parallel-size 2
    • --enable-prefix-caching
    • --enable-auto-tool-choiceS
    • --tool-call-parser qwen3_coder
    • --reasoning-parser qwen3S
    • --max_num_batched_tokens 4096
    • --disable-custom-all-reduce

    Observation: The agent loop may terminate unexpectedly because the returned result does not contain a function call.

  4. ZenoAFfectionate commented on Apr 6, 2026

    @ZenoAFfectionate
    Author

    Yes, it's very strange. Everything works properly when I switch from Qwen3.5 to Gemma4, which shows my code agent framework is fine. I used to think it was my fault!

  5. czerwiakowskim commented on Apr 6, 2026

    @czerwiakowskim

    Have same issue with OpenCode (also appears in Claude Code)

    Image

    this stops agent loop in random moments. Appears despite if using qwen3_coder or qwen3_xml tool parser

  6. dcolt commented on Apr 6, 2026

    @dcolt

    It's nice to see that I'm not alone in experiencing these issues.

    I've been having them since basically the Qwen3.5-122b model got released. (Running on a DGX Spark). But I had no idea what was causing the issue.

  7. bitbottrap commented on Apr 6, 2026

    @bitbottrap

    Another nice enhancement would be combing "tool_choice=auto" with guided decoding functionality with structural tags.

  8. ZenoAFfectionate commented on Apr 7, 2026

    @ZenoAFfectionate
    Author

    I have a sudden thought: perhaps it's a bug caused by dependency conflicts after upgrading vllm to 0.19.0 to be compatible with Gemma4. This is mainly because I found that for vllm, qwen3.5 depends on the 4.x version of the transformers library, while gemma4 depends on the 5.x version.

  9. czerwiakowskim commented on Apr 7, 2026

    @czerwiakowskim

    @ZenoAFfectionate did you try to replicate on any previous versions of vLLM? Maybe temporary solution for people who need qwen3.5 to work properly is downgrade a bit?

  10. ZenoAFfectionate commented on Apr 8, 2026

    @ZenoAFfectionate
    Author

    @ZenoAFfectionate did you try to replicate on any previous versions of vLLM? Maybe temporary solution for people who need qwen3.5 to work properly is downgrade a bit?

    When Qwen3.5 first came out, it worked fine. I remember the vLLM version back then was 0.17, but after updating to 0.18 and 0.19, it became unusable – that’s what shocked me the most. So maybe you could try downgrading to version 0.17~

  11. czerwiakowskim commented on Apr 8, 2026

    @czerwiakowskim

    @ZenoAFfectionate did you try to replicate on any previous versions of vLLM? Maybe temporary solution for people who need qwen3.5 to work properly is downgrade a bit?

    When Qwen3.5 first came out, it worked fine. I remember the vLLM version back then was 0.17, but after updating to 0.18 and 0.19, it became unusable – that’s what shocked me the most. So maybe you could try downgrading to version 0.17~

    I run vLLM on lates cuda-130:nightly docker image build which is v0.19. Seems like I wont get away from crafting my own docker image for blackwell architecture and 0.17 vLLM.

    BTW: In the meanwhile i've created simple proxy resoling problem with malformed tool calls: https://github.com/czerwiakowskim/qwen-toolcall-fixer it's OpenAI API compatible, so if anyone need a quick fix working for most of the time grab it and use it :)

  12. dcolt commented on Apr 8, 2026

    @dcolt

    Something that seems to be a workaround is to use the anthropic api instead of openai api to communicate with VLLM (works with both Claude Code and OpenCode).

    Through Claude Code I've run it for a couple of days without any issues. And now for a few hours in OpenCode without issues.

    No idea why it would actually make a difference.

  13. epheien commented on Apr 9, 2026

    @epheien

    Same issue in OpenCode.

  14. thad0ctor commented on Apr 10, 2026

    @thad0ctor

    I’m seeing what looks like the same issue on a local Qwen3.5 + OpenClaw setup.

    Environment:

    • vLLM: 0.19.0+cu130
    • model: Qwen3.5-122B-A10B-AWQ
    • client: OpenClaw via OpenAI-compatible /v1/chat/completions

    Server flags:

    • --host 0.0.0.0
    • --port 6001
    • --tensor-parallel-size 4
    • --gpu-memory-utilization 0.975
    • --max-num-seqs 6
    • --max-model-len 262144
    • --reasoning-parser qwen3
    • --enable-auto-tool-choice
    • --tool-call-parser qwen3_coder

    Observed behavior:

    • Qwen3.5 can produce reasoning normally with --reasoning-parser qwen3.
    • Tool calling appears unreliable in the qwen3 + qwen3_coder combination,
      consistent with tool calls ending up inside LLM response
  15. athanasiaorange commented on Apr 10, 2026

    @athanasiaorange

    我也遇到了相同的问题,本地vllm+Qwen3.5-35B,openclaw调用LLM时,一但思考就会中断,导致回答总是原地踏步。

  16. 9 remaining items

  17. added a commit that references this issue on May 26, 2026
  18. bbrowning commented on Jun 12, 2026

    @bbrowning
    Contributor

    We have a complete rewrite of the streaming qwen3 parser (as the first of several to use a new streaming parser engine) in #45413 if anyone from this thread that reported the various permutations of this type of issue wants to give things a try with that one. Building with that PR locally will automatically update your qwen3_coder/qwen3_xml reasoning tool call parsers and qwen3 reasoning parser to the new parser engine.

  19. 0DukePan commented on Sep 7, 2026

    @0DukePan

    Independent confirmation & version-range gap for PR consideration

    The root-cause analysis in this issue is accurate. This isn't specific to the 35B model—the same qwen3 reasoning parser and qwen3_coder tool parser interaction is reproducible across other Qwen3.5 variants as well (including Qwen3.5-27B-AWQ).

    For the fix and corresponding test coverage in PR #39055, it is worth explicitly accounting for the behavior across the >=0.20.0 to <0.24.0 version range, where this is only partially addressed:

    • <=0.19.0: The tool call emitted inside the think block never reaches the tool parser.
    • >=0.20.0 <0.24.0: Earlier parser changes partially resolved chunk routing, but the paired tool_call block pattern described in this report can still be missed.
    • >=0.24.0: Resolved for this specific pattern following the code path rewrite.

    Ensuring test coverage spans the 0.20–0.23 release behavior will prevent regressions between earlier parser iterations and the full fix.

    For reference and CI regression tracking, the reported failure shape (both non-streaming and chunked streaming variants) is tracked here:

    Happy to re-test the proposed PR build against these raw fixture patterns once it is ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions