Eval bug: Qwen3.5 9B often prints tool calls in XML and stops when thinking is enabled - tool calls inside thinking block

Name and Version

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16368 MiB):
Device 0: AMD Radeon RX 6900 XT, gfx1030 (0x1030), VMM: no, Wave Size: 32, VRAM: 16368 MiB
version: 8461 (cea560f)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

HIP

Hardware

CPU: AMD Ryzen 9 5950X
GPU: AMD Radeon RX 6900 XT
OS: NixOS 26.05 (Yarara) x86_64

I think the GGML backend is HIP because that's what the nixpkgs package llama-cpp-rocm enables - https://github.com/NixOS/nixpkgs/blob/a1e8ce6b50ffa87ad0d39881c47eb214982330dc/pkgs/by-name/ll/llama-cpp/package.nix

Models

unsloth Qwen3.5-9B-UD-Q4_K_XL
https://huggingface.co/unsloth/Qwen3.5-9B-GGUF

Problem description & steps to reproduce

I'm running llama.cpp server with the command:

llama-server --port 8001 --models-preset ~/llama.cpp/models.ini --host 0.0.0.0

My models.ini contains:

version = 1

kv-unified = true
cache-type-k = q8_0
cache-type-v = q8_0

[Qwen3.5-9B-UD-Q4-Coding]
model = /home/user/llama.cpp/unsloth/qwen3.5-9b-ud-q4_k_xl/unsloth_Qwen3.5-9B-GGUF_Qwen3.5-9B-UD-Q4_K_XL.gguf
mmproj = /home/user/llama.cpp/unsloth/qwen3.5-9b-ud-q4_k_xl/unsloth_Qwen3.5-9B-GGUF_mmproj-BF16.gguf
alias = Qwen3.5-9B-UD-Q4-Coding

ctx-size = 131072
fit = on
flash-attn = true
chat-template-kwargs = {"enable_thinking":true}

temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.00
presence-penalty = 0.0
repeat-penalty = 1.0

Then I use the Zed editor to connect to my model from another computer. Here's the relevant bits of my chat with it using Zed's feature to export your agent chat to markdown:

Chat markdown

## User

What tools do you have?

## Assistant

<thinking>
The user is asking about what tools I have available. I should provide a clear, helpful summary of the tools I can use, formatted nicely in markdown. I don't need to use any tools for this - I can just explain them from the information I have.

</thinking>

I have access to several tools that can help you work with code and projects. Here's what I can do:

## File Operations

- **`read_file`** - Read the content of files in your project. If files are large, I'll get an outline with line numbers so we can dive into specific sections.
- **`list_directory`** - List the contents of a directory to explore the project structure.
- **`find_path`** - Fast file path matching using glob patterns (e.g., `**/*.rs` to find all Rust files). Great for locating files by name.
- **`grep`** - Search file contents using regular expressions. Prefer this over path search when looking for symbols or specific code patterns.

## Project Diagnostics

- **`diagnostics`** - Get errors and warnings for the project or a specific file. Useful for checking the health of your code after making changes.

## Web & Network

- **`fetch`** - Fetch URLs and return the content as Markdown. Handy for loading documentation or external resources.
- **`open`** - Open files or URLs with your default application (browser, text editor, etc.).

## System Tools

- **`now`** - Get the current datetime in RFC 3339 format (UTC or local time).

## Advanced

- **`spawn_agent`** - Delegate tasks to agents when you need parallel work or complex tasks that benefit from a dedicated session.

---

All of these tools are designed to work with your project in the root directory `/path/to/project`. Just let me know what you're working on and I can help you navigate and manipulate the code!

## User

Read through [@main.nu](file:///path/to/project/main.nu) and give me a summary of what it does

## Assistant

<thinking>

I need to read the actual content of main.nu file to provide a summary - the file outline only shows 7 symbols which is not enough detail.

</thinking>

I can see the file outline shows this is a Nushell script with some notes about <omitted>. However, I only see a file outline with symbol names. Let me read the full content to give you a comprehensive summary.



**Tool Call: Read file `main.nu`**
Status: Completed

```main.nu
SUCCESS: File outline retrieved. This file is too large to read all at once, so the outline below shows the file's structure with line numbers.

IMPORTANT: Do NOT retry this call without line numbers - you will get the same outline.
Instead, use the line numbers below to read specific sections by calling this tool again with start_line and end_line parameters.

# File outline for /path/to/project/main.nu

<omitted>

Showing symbols 1-7 (total symbols: 7)


NEXT STEPS: To read a specific symbol's implementation, call read_file with the same path plus start_line and end_line from the outline above.
For example, to read a function shown as [L100-150], use start_line: 100 and end_line: 150.
```


## Assistant

<thinking>

The file is too large and I only see a structure outline. I need to read specific sections. Let me start from the beginning to understand the overall structure.

<tool_call>
<function=read_file>
<parameter=path>
main.nu
</parameter>
<parameter=start_line>
1
</parameter>
<parameter=end_line>
100
</parameter>
</function>
</tool_call>
</thinking>

After the agent prints the tool call in XML it simply stops running as if it's finished.

Sometimes it succeeds making one tool call, but then it starts printing XML and I give up on it. I have no idea if it may or may not make another successful call if you try enough times after it starts printing XML.

However, if I disable thinking with chat-template-kwargs = {"enable_thinking":false}, then it makes all its tool calls without problems (at least as far as I've tested), e.g. it can read the whole 1000-line file in several consecutive tool calls.

I've also tried running llama-server with the flag --no-cache-prompt as mentioned in #20614 but that had no effect.

I think the same or similar issue is also reported here
ollama/ollama#14745
ollama/ollama#14493

First Bad Commit

I've had this problem since llama.cpp 8255. I haven't tried earlier versions.

Relevant log output

Here's the log from llama-server from running the model until the XML tool call. It shows that it's served several requests because the model actually tried to read the file multiple times but it got the path wrong. When it finally got it right, that's when it printed the XML.

log.txt

This issue might be similar or related to the following issue(s):

#[20809]: Qwen3-Instruct-2507 tool calling broken by false thinking detection (tool calls captured in reasoning_content)
#[20265]: Model output directed to Reasoning block instead of standard response
#[20260]: peg-native parser fails when model outputs text before thinking tag (tool calling + thinking model)
#[20833]: Qwen3.5-2B thinks by default (similar Qwen3.5 thinking mode issues)
#[20409]: enable_thinking=false via --chat-template-kwargs is ignored (Qwen3.5 thinking toggle issues)
#[20182]: enable_thinking param cannot turn off thinking for qwen3.5
#[20789]: tags forcefully injected despite reasoning_format=none
#[20754]: Autoparser misclassifies all output as reasoning for templates with /no_think toggling
#[20325]: Autoparser misplaces non-thinking content with NVIDIA-Nemotron-Nano-9B-v2
#[20008]: Cannot get reasoning_content with Kimi K2.5 (reasoning block handling)
#[20516]: Response always starts with tag when running Qwen3.5 9B
#[20196]: disabling reasoning does not work anymore on certain models
#[20458]: reasoning off gives reasoning medium for gpt-oss
#[20717]: DeepSeek V3.2 no longer reasons when using jinja template
#[20353]: Gets stuck in a loop while thinking

This comment was auto-generated locally using llama.cpp version: 8456 (b31b30f) on NVIDIA DGX Spark

Could it be that the problem only happens when a tool call is made inside a thinking block?

Yes, this is an issue that I'll be working on now that #20424 is merged. The problem is that the grammar trigger is independent from the reasoning parser and the grammar forces <eog> when it's done reading what it considers a tool call, so the model can't generate further (it should close the thinking block and redo the tool call properly within the normal content block).

I think I’m hitting the same underlying bug, but through a different stack.

My setup:

Claude Code
LM Studio on Windows
qwen/qwen3.5-35b-a3b
thinking enabled

What I’m seeing:

plain /v1/messages text responses work
streaming works
simple tool-use requests can work
but in longer agentic/tool-heavy sessions, the model sometimes places a fake XML-like tool call inside the thinking block instead of returning a real structured tool_use block
the response then ends with stop_reason: "end_turn", so the client treats it as finished

So from the client side it looks like the model decides to use a tool, but only describes the tool call inside thinking rather than emitting an actual tool call payload.

This seems very similar to the issue described here, especially the “tool calls inside thinking block” behavior.

Yes, this affects all models that have a tendency to add tool calls within thinking blocks, either by mistake or by "preparing" a tool call in thinking.

This behavior manifests across all models in the Qwen3.5 family, although it seems very particularly common on 35B-A3B and 122B-A10B. As others have noted, once it hits </tool_call> (even though it's weirdly inside the reasoning block, output stops.

As far as I understand the parsing architecture of llama.cpp, even if the model continued generating, the tool call would never get parsed and returned as an actual tool call, and handling it properly in llama.cpp's parser would require a bit of a rewrite to the flow, or specific handling logic for Qwen3.5 models, since the tool-call parsing is invoked after (and separately from) reasoning block parsing.

I was considering doing a PR for handling this, but realizing that it wasn't really possible to fix surgically, I resorted to adding a "quirks handling" system in Humanity's Last Command, with a specific quirk handler for Qwen3.5 that fixes the issue locally.

Technically, it's quite easy to handle, and you can probably apply the "tool thought" handler to just about any downstream app, if it has similar functionality to lc's quirks handling.

Having proper handling directly in llama.cpp would be pretty sweet though, and if maintainers give the thumbs up, I'd consider adding a PR to handle it.

But honestly, I think it's questionable whether such weird model behavior should actually be "fixed" inside llama.cpp's parsing logic, since it's kinda breaks the existing assumptions of the parsers.

@markqvist

@markqvist try PR above

with a specific quirk handler for Qwen3.5 that fixes the issue locally.

This implementation and the one in vLLM will emit any tool calls in the reasoning as a real tool call (feel free to correct me if I'm wrong). I took a look at the sglang implementation and it doesn't appear to do the same. The reasoning parser will consume those tokens, so the tool calling parser never sees them. I'm not 100% sure, but I assume this is the reference implementation and one we should adopt.

I stuck with the same issue i guess. Zed + llama.cpp (b8547, b8548 either not yet build or missing file for download). It just stops work, if you ask again, result in the same attempt to make tool_call but nothing happened. I can provide more info if needed, just ask, because it's matter of 1-2 minutes to reproduce this one.

unsloth\Qwen3.5-35B-A3B-UD-IQ4_NL.gguf (also was on Q4_K_M)
upd: i also got this bug on GLM4.7-Flash, but it was after like 5-7 minutes. On Qwen model it appears very fast, which is sad, because this is the best model that fits consumer hardware.

What version don't have this regression? I saw on Reddit that b8227 has new parser where this bug was introduced, but i rollback to one lower version and still has errors (different ones, works even worse).

Add interleaved_reasoning option to openai compatible models (zed-ind…

@mikecoker

Custom jinja template test results on Qwen3.6-35B-A3B

Model: Qwen3.6-35B-A3B (APEX I-Compact, 17GB, 8GB VRAM)
llama.cpp: b9536 (308f61c)
Template: qwen-3.5-35b.jinja.txt by @mikecoker from this thread
Server: --jinja --chat-template-file qwen-thinking-fix.jinja

Test Results

Scenario	Simple EN (no tools)	Complex CN (with tools)
Reasoning	0 chars	0 chars
Content	✅ Non-empty, includes thinking text	⚠️ 18K chars thinking text, no tool call
Tool executed	N/A (simple prompt)	❌ No tool called
Name fidelity	✅ Correct	❌ Hermes→hermet(78x)/hermos/trail
Time	6-17s	34-207s
Finish	stop/length	stop

Analysis

The template correctly redirects thinking from reasoning_content to content — reasoning_content is always 0 chars
For simple prompts (no tools), output is clean and usable
For complex tool-calling prompts, the model generates long thinking chains as content text but never executes the tool call, and name degradation appears
Same core issue as Eval bug: Qwen3.5 9B often prints tool calls in XML and stops when thinking is enabled - tool calls inside thinking block #20837 persists: the model generates tool calls during thinking, the parser doesn't recognize them, and the model loops

Working Workaround

chat_template_kwargs: {"enable_thinking": False} per-request completely resolves all issues:

0 reasoning, 0 stall, tool executed, name correct, 5s response time

Version Info

version: 9536 (308f61c31)
built with Clang 19.1.5 for Windows x86_64

This was tested on Qwen3.6 (not Qwen3.5). The template may work better on Qwen3.5 as confirmed by @heipei.

Test results: `--reasoning-format none` + thinking ON on Qwen3.6-35B-A3B

Model: Qwen3.6-35B-A3B (APEX I-Compact, 17GB, 8GB VRAM)
llama.cpp: b9536 (308f61c)
Server: --jinja --reasoning-format none --spec-type draft-mtp --temp 0.9
Test: complex Chinese tool-calling prompt (write_file + path + content)

We tested 3 temperatures × 3 runs each with thinking ON (no chat_template_kwargs override).

Results

Temp	R1	R2	R3	Perfect rate	Avg time
0.6	✅ 99s	⚠️ 101s (11 corruptions)	⚠️ 81s (1 corruption)	33%	93.6s
0.8	⚠️ 78s (5 corruptions)	✅ 33s	✅ 34s	67%	48.3s
0.9	⚠️ 174s (103 corruptions)	✅ 12s	⚠️ 42s (3 corruptions)	33%	76.0s

"Perfect" = write_file executed + correct path + correct filename + correct content + zero name corruptions + no thinking tags visible in output.

Key findings

--reasoning-format none makes thinking ON + tool calling work — tool calls are no longer swallowed by the reasoning parser. Content is always non-empty.
temp 0.8 is the sweet spot — 67% perfect rate, fastest average time at 48s.
Name degradation still occurs — when the model generates long thinking chains (~103 corruptions at 0.9 R1), name fidelity suffers. This appears correlated with chain length, not temperature directly.
Thinking tags are NOT visible in content — contrary to the help text description ("leaves thoughts unparsed in message.content"), the output is clean.

Comparison with thinking OFF

Mode	Perfect rate	Avg time	Thinking preserved
thinking OFF	73% (8/11)	3.4s	❌
reasoning-format none + ON	44% (4/9)	72s	✅

Trade-off

--reasoning-format none is a viable workaround for thinking ON + tool calling. It's significantly slower and less reliable than disabling thinking entirely, but it preserves the model's reasoning capability for tasks that benefit from it.

Tested on Qwen3.6 (not Qwen3.5). Would be interesting to see if results differ on other versions.

Regression report: b9933 `--mlock` failure and 70% speed drop on 8GB VRAM + Q4_K_M

Model: Qwen3.6-35B-A3B-UD-Q4_K_M (22GB, 8GB VRAM, RTX 4060)
llama.cpp: b9933 (92366df, built with Clang 20.1.8)
Test: prompt processing + generation throughput benchmark

Observations

1. --mlock fails with "system resources insufficient"

SetProcessWorkingSetSize failed: 系统资源不足，无法完成请求的服务

This happens consistently across reboots. On b9536, --mlock works fine with the same model and hardware.

2. Generation speed drops 70-78% across all modes

Metric	b9933	b9536 (baseline)	Regression
Gen	12.9 t/s	57.8 t/s	-78%
Short	9.3 t/s	35.1 t/s	-74%
Long	9.0 t/s	29.7 t/s	-70%
Agent	7.1 t/s	30.5 t/s	-77%
RealSim	timeout	14.7 t/s	❌

3. GPU stuck at low power state
During inference: GPU clock 1890 MHz / 16W (full: 2625 MHz / 47W). Only 1/3 of expected power. The Bubbles screen saver wake trick doesn't help. b9536 runs at full speed on the same hardware.

Analysis

The root cause appears to be a combination of:

--no-mmap consumes more VRAM in b9933 → RealSim OOM
--mlock unavailable → CPU page faults slow down inference
CPU bottleneck prevents GPU from reaching full clock → stuck at 16W

Most changes in b9933 are in Vulkan/OpenCL/WebGPU backends, unrelated to CUDA. The regression likely comes from a build/linking change (Clang 20.1.8?) or an unintended side effect in the CUDA path.

Workaround

Stay on b9536. No known workaround for b9933 on this configuration.

Hardware: RTX 4060 8GB, Intel Ultra 5 125H, Windows 11
llama.cpp: b9933 (92366df)

Update: confirmed APEX I-Compact (17GB) also affected

We tested a second model to isolate the root cause. APEX I-Compact (17GB, fits entirely in 8GB VRAM with room to spare) on b9933:

Model loaded successfully
--mlock works (only fails on 22GB Q4_K_M in 8GB VRAM)
Gen benchmark cold run exceeded 5 minutes (should complete in 7-8 seconds)
GPU stuck at 1890 MHz / 16.88 W (full speed: 2625 MHz / 47W)
Same GPU non-boost behavior as Q4_K_M

This confirms the regression is NOT specific to Q4_K_M's VRAM pressure:

Scenario	b9536	b9933
Q4_K_M (22GB) + --mlock	Gen 57.8	--mlock fails
Q4_K_M (22GB) no --mlock	--	Gen 13, GPU 16W
APEX (17GB) + --mlock	--	cold run 5min+, GPU 16W

b9933 prevents the GPU from boosting (1890 MHz ceiling instead of 2625 MHz) regardless of model size. Likely a CUDA driver interaction issue introduced between b9536 and b9933.

Recommendation: Stay on b9536 until this is resolved.

Update: Enhanced chat template tested on llama.cpp with Qwen3.6-35B-A3B (8GB VRAM)

After our reports on --reasoning-format none, we tested the community qwen3.6-enhanced.jinja template from the vLLM ecosystem.

The template (allanchan339's qwen3.6-enhanced.jinja) has excellent features:

Self-healing logic: auto-closes unclosed thinking tags before <tool_call>
Clean tool_call formatting: proper XML wrapping
enable_thinking=true/false support: handles both modes

However, on 8GB VRAM with llama.cpp + APEX I-Compact (17GB):

When thinking=ON + the template's thinking prefill:

Tool execution: 100% (better than default template's 50%)
But the prefill forces the model into a long thinking chain (34-90s)
Long thinking causes name degradation in the 3B active-parameter MoE (Hermes trial → Hermes trail, 1-6 corruptions per run)
The model's 3B active parameter budget can't maintain name fidelity across 5000+ tokens of thinking

When thinking=OFF (enable_thinking=false):

The template still outputs an empty thinking\n\n response\n\n block in the generation prompt
This confused the model and dropped perfect rate from 73% (default template) to 22%

Conclusion for llama.cpp users with limited VRAM:

--reasoning-format none alone (no custom template) gives the best results
The vLLM ecosystem's template solutions don't translate well to llama.cpp's template architecture
The fundamental bottleneck on low-VRAM setups is the model's active parameter count (3B), not the template format
PR Add specialized tagged thinking tool parser #24202 (tagged thinking tool parser) is still the most anticipated fix

Also regarding b9933:

We tested b9933 on RTX 4060 8GB and found severe regressions:

--mlock fails with "system resources insufficient" on 22GB Q4_K_M (works on b9536)
GPU stuck at 1890 MHz / 16W (full speed 2625 MHz / 47W) regardless of model size
Generation speed dropped 70-78% across all benchmarks
Full details at Eval bug: Qwen3.5 9B often prints tool calls in XML and stops when thinking is enabled - tool calls inside thinking block #20837 (comment)

Hope this helps others with similar hardware constraints.

Follow-up: Bug 123 resolved with Ornith-1.0-35B (RL-optimized Qwen3.6 variant)

After extensive testing on Qwen3.6-35B-A3B (8GB VRAM, llama.cpp b9536), I identified three core bugs with thinking ON + tool calling:

Bug	Description	Root cause
Bug 1	Tool calls swallowed into reasoning_content (content="")	Parser doesn't recognize tools inside thinking blocks
Bug 2	Name degradation (Hermes→Helmes/Hermos/trail)	3B active params drift in long thinking chains
Bug 3	Infinite reasoning loops (45K+ tokens, no tool call)	MTP + thinking conflict

Tested mitigations (no full fix):

--reasoning-format none → fixes Bug 1 but not 2 or 3
Custom jinja templates → partial Bug 1 fix
Temperature tuning (T0.5 best) → reduces Bug 3 frequency but doesn't eliminate
MTP disabled → helps Bug 3 but doesn't fix Bug 1/2

Ornith-1.0-35B changes the picture

Ornith-1.0-35B is an RL-optimized variant of Qwen3.6-35B (Terminal-Bench 64.2 vs 52.5). Tested with multi-turn interactive framework (max 5 turns, tool execution + result return):

Metric	Qwen3.6-35B (NOMTP)	Ornith-1.0-35B
Bug 2 (degradation)	1~6 per run	0 ✅
Bug 1 (content empty)	1~3/3 at T0.8	0/3 at T0.8 ✅
Bug 3 (infinite loop)	Common at T0.8	None observed ✅
Write_file success	~50% multi-turn	~89% multi-turn ✅
Format consistency	2 formats	7+ formats ⚠️

Key limitation

Ornith uses 7+ different tool call formats (XML <write_file> with/without closing tags, JSON arrays [{"name":"write_file","arguments":{}}], JSON objects {"path":"..."}, python-style write_file(path=...)). This suggests the RL training exposed it to inconsistent tool-calling data. The model always gets the content right (path, filename, zero degradation) but varies how it expresses the call.

Bottom line

Zero name degradation is Ornith's killer feature — the 3B active params can maintain precision through thinking chains
Format inconsistency is the trade-off from RL optimization
For practical use: Ornith at T0.5/T0.6 with a robust tool-call parser works well

Regression report: b10042 CUDA speed regression on 8GB VRAM (RTX 4060)

b10042 升级回归报告（2026-07-16）

概述

b10042（2026-07-16 升级）→ 全线速度异常。已回滚至 b9536。

速度对比

Ornith moe28

指标	b9536 ✅	b10042 ❌	降幅
启动	55s	82s	+49%
Gen	74.3	—（未完成）	—
Short	37.7	~7 t/s	-81%
Long	38.0	~13 t/s	-66%
Agent	31.4	—	—
RealSim	34.7	—	—

APEX moe33

指标	b9536 ✅	b10042 ❌	降幅
启动	33s	128s	+288%
Short	42.2	全部 cancel ❌	—
Long	31.5	全部 cancel ❌	—
Agent	38.2	全部 cancel ❌	—
RealSim	31.7	全部 cancel ❌	—

根因分析

1. b10042 的 CUDA 实现变小了

文件	b9536	b10042	变化
ggml-cuda.dll	154 MB	140 MB	-9%
llama-server-impl.dll	15.3 MB	9.7 MB	-37%

CUDA 核心库缩小 9%，可能移除了部分 GPU kernel 优化，或改用了更保守的 CUDA graph 编译策略。

2. 任务全被 cancel 的原因

不是新 bug——是每个请求的处理速度骤降，导致 Python benchmark 脚本的 HTTP 超时（120s/300s）被触发，客户端断开连接后 server 记录 "cancel task"。

b9536：Short 37.7 t/s → 50 tok 在 ~1.3s 完成
b10042：Short ~7 t/s → 50 tok 在 ~7s 完成（慢 5x，但仍在超时内）
b10042 APEX：每个 task 耗时远超超时 → 客户端断开 → cancel

3. tg_3s 新指标

b10042 新增 tg_3s 字段（3 秒窗口内的生成速度），说明新版改变了任务时序监控机制，可能与速度回归有关。

结论

b10042 在 8GB VRAM 场景下不可用。 ggml-cuda.dll 缩小 9% 后，笔记本 GPU 的 CUDA graph 性能大幅退化。已回滚至 b9536。

回滚验证：./bin-b9536/llama-server.exe --version → version: 9536

Add interleaved_reasoning option to openai compatible models (zed-ind…

Correction: the fix was removing DRY, not Ornith itself

In my previous report (comment #4958517549) I said "Bug 123 resolved with Ornith-1.0-35B". This was misleading — the real fix was removing DRY parameters, not switching to Ornith.

After further cross-model testing, the actual findings are:

Model	DRY?	Degenerations	Path correct	WF rate
Qwen3.6 APEX I-Compact	ON (0.75/2.0/2048)	179~334 ❌	11%	3/3
Qwen3.6 APEX I-Compact	OFF	0 ✅	100%	3/3
Qwen3.6 UD-Q4_K_M-nomtp	ON	2~5 ❌	0%	2/3
Qwen3.6 UD-Q4_K_M-nomtp	OFF	0 ✅	100%	3/3
Ornith-1.0-35B	OFF (no DRY in config)	0	100%	3/3

Ornith just happened to not have DRY in its config. When we removed DRY from the Qwen3.6 configs, they performed identically well — all three bugs (empty content, name degeneration, infinite loop) and path transcription errors were 100% resolved.

See the full DRY analysis on #15066:

Eval bug: DRY sampling appears to have no effect #15066 (comment) (DRY parameter tuning)
Eval bug: DRY sampling appears to have no effect #15066 (comment) (path transcription side effect)
Eval bug: DRY sampling appears to have no effect #15066 (comment) (cross-model validation)

Takeaway: DRY with thinking ON is actively harmful for tool-use scenarios across all tested models. The trade-off between repetition prevention and path/tool degeneration is not worth it when thinking is enabled.

i have this situation of xml tool calling in thinking mode very often with qwen 3.6-27B. I have no DRY:


    --no-mmproj \
    --jinja \
    `# --- Thinking Mode ---  --reasoning-budget 0  --help` \
    --reasoning on \
    --reasoning-preserve \
    `# --- Server Setup & Logging ---` \
    --host 127.0.0.1 \
    --port 8080 \
    --api-key pi-local \
    --no-ui \
    --metrics \
    --log-verbosity 4 \
    `# --- Hardware, Compute & Threads ---` \
    --n-gpu-layers all \
    --flash-attn on \
    --threads 1 \
    --threads-batch 1 \
    `# --- Context & Batching ---` \
    --ctx-size 131072 \
    --batch-size 1024 \
    --ubatch-size 1024 \
    --fit on \
    --no-context-shift \
    `# --- KV Cache & Slots (Concurrency) ---` \
    --parallel 1 \
    --cache-prompt \
    --cache-ram 0 \
    --no-cache-idle-slots \
   `# --- KV Cache Debugging ---` \
   --slot-prompt-similarity 0.1 \
    `# --- GDA/Recurrent Context Checkpoints ---` \
    --ctx-checkpoints 4 \
    --checkpoint-min-step 4096 \
    `# --- Speculative Decoding (MTP) ---` \
    --spec-type draft-mtp \
    --spec-draft-n-max "${draft_tokens}" \
    `# --- Sampling Parameters ---` \
    --samplers 'top_n_sigma;top_k;typ_p;top_p;min_p;temperature' \
    --top-n-sigma 3 \
    --top-k 20 \
    --typical-p 0.99 \
    --top-p 0.95 \
    --min-p 0.02 \
    --temperature 0.9 \
    --dynatemp-range 0.0 \
    --mirostat 0 \
    --no-warmup \
    --prio -1 \
    --poll 0 \
    --poll-batch 0

3 turns isn't proof of a fix, at all. When it does 30 turns, and is 100k into context, and doesn't go off in the weeds, I'll consider it worth my time.
ALL these llm's even on the big-boy clouds screw up and go off into the weeds.
IMHO, it's not the harness, not llama.cpp, it's crummy RL/HRL and poor tuning before release.
My issue is that there's always one part that's broken.
Tool calls that don't go back to reasoning is a huge deal breaker, messes up workflows. Unreliable broken toys.
for one-off summaries, or poetry, fine. but not real actual work, for that you need a human.

I reproduced this on the exact bug-era build and model from this report (b8461 / cea560f, unsloth/Qwen3.5-9B-UD-Q4_K_XL GGUF) with thinking enabled (this model's chat-template default). In a multi-turn agent loop it did precisely what's described here:

Turn 1: A list_directory tool call is parsed and executed successfully.
Turn 2: The model emits the XML envelope for a read_file call (<tool_call><function=read_file>...) inside the thinking block, then stops — finish_reason: stop, with tool_calls entirely absent from the response message.
The same failure occurs in both non-streaming and streaming modes.

Observation for a potential fix:
It appears the server splits the output on the closing </think> tag and routes everything before it into reasoning_content — while downstream tool-call parsing only inspects content. As a result, a syntactically complete XML tool call inside the thinking block never reaches the tool parser, and the client receives a well-formed 200 response that looks as if "the model chose not to act." This also explains why single-turn prompts mostly work: the issue appears when the model starts planning its next call inside the thinking block after an earlier success.

I captured the raw provider responses (pre-processing) for both non-streaming and streaming failures. The reproduction protocol and full evidence are available here:

Happy to attach either raw response directly to this thread if needed.

chat : allow tool calls inside reasoning blocks for Ministral-3 (ggml…

Eval bug: Qwen3.5 9B often prints tool calls in XML and stops when thinking is enabled - tool calls inside thinking block #20837

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

github-actions commented on Mar 21, 2026

kik4444 commented on Mar 21, 2026

pwilkin commented on Mar 21, 2026

arthuz-th commented on Mar 21, 2026

pwilkin commented on Mar 21, 2026

markqvist commented on Mar 22, 2026

pwilkin commented on Mar 22, 2026

aldehir commented on Mar 22, 2026

heavycharged commented on Mar 27, 2026

43 remaining items

Sensenkawa commented on Jul 8, 2026

Custom jinja template test results on Qwen3.6-35B-A3B

Test Results

Analysis

Working Workaround

Version Info

Sensenkawa commented on Jul 8, 2026

Test results: --reasoning-format none + thinking ON on Qwen3.6-35B-A3B

Results

Key findings

Comparison with thinking OFF

Trade-off

Sensenkawa commented on Jul 9, 2026

Regression report: b9933 --mlock failure and 70% speed drop on 8GB VRAM + Q4_K_M

Observations

Analysis

Workaround

Sensenkawa commented on Jul 9, 2026

Sensenkawa commented on Jul 9, 2026

Update: Enhanced chat template tested on llama.cpp with Qwen3.6-35B-A3B (8GB VRAM)

The template (allanchan339's qwen3.6-enhanced.jinja) has excellent features:

However, on 8GB VRAM with llama.cpp + APEX I-Compact (17GB):

Conclusion for llama.cpp users with limited VRAM:

Also regarding b9933:

Sensenkawa commented on Jul 13, 2026

Follow-up: Bug 123 resolved with Ornith-1.0-35B (RL-optimized Qwen3.6 variant)

Tested mitigations (no full fix):

Ornith-1.0-35B changes the picture

Key limitation

Bottom line

Sensenkawa commented on Jul 16, 2026

Regression report: b10042 CUDA speed regression on 8GB VRAM (RTX 4060)

b10042 升级回归报告（2026-07-16）

概述

速度对比

Ornith moe28

APEX moe33

根因分析

1. b10042 的 CUDA 实现变小了

2. 任务全被 cancel 的原因

3. tg_3s 新指标

结论

Sensenkawa commented on Jul 18, 2026

dumitru-nicolae-marasoiu commented on Aug 10, 2026

xxxajk commented on Aug 10, 2026

0DukePan commented on Sep 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Test results: `--reasoning-format none` + thinking ON on Qwen3.6-35B-A3B

Regression report: b9933 `--mlock` failure and 70% speed drop on 8GB VRAM + Q4_K_M