Repository navigation
[Bug] GLM-5.3-Flash thinking output degenerates into repeated '!' under multi-tool agentic prompts #36669
Description
Summary
When serving zai-org/GLM-5.3-Flash (native FP8 checkpoint) with SGLang, requests with a complex system prompt + multiple tools + tool_choice=auto cause the model's reasoning/thinking output to degenerate into an endless stream of ! characters. Simple requests work correctly.
Environment
- Image:
lmsysorg/sglang:glm-5.3-flash - SGLang:
0.0.0.dev1+gd6ab04bdf1 - Hardware: 8x NVIDIA H20 (Hopper sm90), TP=8, EP=8
- Model:
zai-org/GLM-5.3-Flash(native FP8)
Launch args:
python3 -m sglang.launch_server --model-path /model \
--tp-size 8 --ep-size 8 \
--kv-cache-dtype bfloat16 \
--dsa-prefill-backend tilelang --dsa-decode-backend tilelang \
--moe-runner-backend deep_gemm --disable-shared-experts-fusion \
--reasoning-parser glm45 --tool-call-parser glm47Repro
Simple request (works fine):
curl http://127.0.0.1:8001/v1/chat/completions \
-H "Content-Type: application/json" -d '{
"model": "/model",
"messages": [{"role": "user", "content": "1+1=? answer with a number only"}],
"max_tokens": 100
}'
# -> content "2", reasoning content is normalComplex request (degenerates):
curl http://127.0.0.1:8001/v1/chat/completions \
-H "Content-Type: application/json" -d '{
"model": "/model",
"messages": [
{"role": "system", "content": "You are a coding agent. Be concise. Use tools when needed. You have bash, read, write, edit, glob, grep tools available... (long agentic system prompt ~2000 tokens)"},
{"role": "user", "content": "1+1=? answer with a number only"}
],
"max_tokens": 200,
"tool_choice": "auto",
"tools": [
{"type":"function","function":{"name":"bash","description":"run shell","parameters":{"type":"object","properties":{"command":{"type":"string"}},"required":["command"]}}},
{"type":"function","function":{"name":"read","description":"read file","parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}},
{"type":"function","function":{"name":"write","description":"write file","parameters":{"type":"object","properties":{"path":{"type":"string"},"content":{"type":"string"}},"required":["path","content"]}}}
]
}'Observed behavior
The streaming response's reasoning_content field contains only ! characters (hundreds of them, e.g. 1800+ ! in a single response), and content is empty. The model never produces a real answer.
What I've already tried
--reasoning-parser glm45-> degenerates (reasoning_content=!!!!!)--reasoning-parser deepseek-r1-> degenerates (reasoning_content=!!!!!)- No reasoning-parser ->
<think>block leaks intocontent(reasoning not stripped) - Disabling speculative decoding (MTP) entirely -> still degenerates (rules out MTP as the cause)
chat_template_kwargs {"reasoning_effort": "low"}-> still degenerates
Notes / hypothesis
- GLM-5.3-Flash thinking is always-on (the chat template injects
<think>unconditionally), controlled byreasoning_effort(low/high/max) rather thanenable_thinking. - Simple requests (short prompt, no tools, or a single simple tool) work correctly.
- The degeneration is triggered specifically by complex system prompt + multiple tools + tool_choice=auto.
- This may be related to how the reasoning parser interacts with tool-call parsing (
glm47) for this model — the model appears to enter a repetition loop once it must reason about multiple tool definitions. - Related: [Bug] NEXTN/MTP speculative decoding fails to load MTP MoE weights under TP>1 for Glm5NextForConditionalGeneration (GLM-5.3-Flash) #36653 (MTP under TP>1) is a separate issue; this degeneration happens with MTP disabled too.
Reacted by Swipe4057
Activity
RenhaoPan commentedon Aug 27, 2026
same issue,GLM-5.3-flash thinking mode is alway on；
"chat_template_kwargs": { "thinking": False } can't effect
handsome-chips commentedon Aug 28, 2026
same problem~
provider: the reply guard reads a kilobyte, watches the thinking, and…
b8zhong commentedon Sep 3, 2026
Hi, can you please test #36143 @handsome-chips @RenhaoPan @ech0potato if your problem is on SM90?
[Feature] Add GLM-5.3-Flash tool call detector (glm53)
Lei-010 commentedon Oct 8, 2026
Cross-platform data point from vLLM (not SGLang), relevant to your note that
reasoning_effort: lowstill degenerates: check what your template's default branch does — on the Aug-26 HFchat_template.jinja, anyreasoning_effortnot in['low','high']silently clamps tomax, so unset/medium requests all run at max effort.Our environment (for context):
v0.30.1rc1.dev539nightly + flashinfer0.7.1rc2, container imagevllm/vllm-openai:v0.30.0-fix14glm45, tool-call parserglm47reasoning_effort(chat_template.jinja dated Aug 26:effective_reasoning_effort = reasoning_effort if reasoning_effort in ['low','high'] else 'max'— note anything not low/high, incl.medium, silently falls back tomax)On vLLM 0.30.1 nightly we measured (synthetic 50-60 turn runs, 15K-token inert prefix, complex tasks, temp=0):
medium)Your "! flooding" reproduction with
reasoning_effort=low(if the value actually reached the template) would suggest a second, effort-independent degeneration path — consistent with the accumulated-decode state issue isolated in vllm#56868 (~200K accumulated decode tokens there). Our runs max out ~60K accumulated decode tokens and never hit the!-style collapse on either stack, so we can't discriminate yet.Also noting for completeness:
enable_thinking/chat_template_kwargs.enable_thinkingis a no-op on this template — onlyreasoning_effort(top-level or as chat_template_kwarg) gates thinking.MMuzzammil1 commentedon Oct 14, 2026
@ech0potato have you faced something similar with GLM-5.2-FP8 Model? I'm facing a similar issue, so kind of wondering if they have the similar origins. 🤔
linkedlist771 commentedon Oct 15, 2026
I have used this in the ZCode, which turns out fine even in the 1M long context.
ByteSage404 commentedon Oct 16, 2026
I have the same problem with self-hosted AI. It errors out for complex tasks. OpenAI is running fine. GLM 3.5 Flash is running on two Nvidia sparks.
Swipe4057 commentedon Oct 16, 2026
glm5.3 flash – issue in 'main' as of today
yibol9768-alt commentedon Oct 17, 2026
Data point from 8x H20: I could reproduce the all-
!thinking output exactly, but only with a corrupted copy of the checkpoint. The bad copy had the same file sizes as the official one, but all 62 sha256 hashes differed and the layer-45 MTP expert weights contained NaN. With that copy, greedy decoding gives!!!!...(token 0) and every logprob is -ln(154880) = -11.95, i.e. the logits are degenerate.With the official weights (sha256 verified), SGLang main (
ed8c3139f) with the launch flags from this issue was clean. That covered complex system prompts (2k and 6.4k tokens) with 3 tools andtool_choice=auto, multi-turn tool-result conversations, with and without EAGLE, about 500 requests, and none degenerated.Details and the official hashes are in #43085 (comment). Quick test that needs no hashing: send a greedy
/generatewith"return_logprob": true. If every logprob is about -11.95, re-download the weights and verify them before debugging the server.Lei-010 commentedon Oct 24, 2026
A data point against the corrupted-checkpoint explanation, for the report this issue came from:
Our GLM-5.3-Flash deployment that produced the degenerate output was verified against the official HF LFS hashes (
zai-org/GLM-5.3-Flash): all 63 files (62 shards + tokenizer) match, 328.4 GB in total, 0 mismatched / 0 missing / 0 extra shards — so at least in our case the weights were not silently corrupted, and the!!!!…degeneration has to be explained by something other than a bad copy of the checkpoint.Given that, the remaining suspects in our setup are the SM120 paths themselves rather than the files: native sparse-MLA (
FLASHINFER_MLA_SPARSE_SM120) plus MTP/speculative decoding, where we also see draft-acceptance collapse (0% for several k values). If anyone has a clean-weights repro that still degenerates, that narrows it to the model runner rather than the checkpoint, which is the more useful report for this thread.justicel commentedon Nov 6, 2026
A data point from our deployment that might help separate the causes here:
We see the same all-
!output, but as a persistent engine-state problem rather than a per-request one. After a single ~1M-token request on v0.5.23 (also reproduced on the 20261028 nightly, where it degenerates inside that first request), every subsequent request on the pod degenerates — empty content or the same!wall — MTP accept rate drops to 0.00, grammar-constrained requests start 500ing, and/flush_cachedoesn't clear any of it. A restart does. v0.5.22 is clean under the same test: 7 more ~1M requests across all three replicas today (fresh prompt each time), all correct, accept rate 0.54-0.74 after each round.The NaN-in-layer-45 observation above caught my eye, because layer 45 is the NextN draft layer — and the commit I suspect most in the v0.5.22 → v0.5.23 window (#44098, which shrinks the KDA state pool from 35 to 34 slots per buffer) does so on the premise that the NextN layer never reads or writes KDA state. We run
speculative_attention_mode=prefill, so the draft does execute during prefill. If that premise is wrong on some path, the pool shrink turns into an out-of-bounds write next to the pool, which would fit the "garbage state until restart" behavior we see.Full repro and config in #48964.