Qwen3.8 Claude Code

Introduction

This repository is the follow-up to muse-glimmer-claude-code. That project ran Meta's Muse-Glimmer-30B on a MacBook Pro (M4 Max, 36GB) and connected it to Claude Code through LiteLLM. This repository does the same with Alibaba's Qwen3.8-27B, using the same llama.cpp build and the same benchmark cases so the results are directly comparable.

It also answers a second question. Unsloth publishes a 1-bit quant of this model at 6.2GB, small enough for an 8GB machine, with a claimed ~77% accuracy retention, and that claim has been doing the rounds as "a frontier-class model on your laptop". So both tiers are measured here side by side: the 1-bit and the 4-bit, on the same hardware, the same benchmarks, and the same agentic test.

Both Qwen3.8-27B and Glimmer-30B are reasoning models with built-in speculative decoding (MTP here, dflash there). Qwen3.8-27B also has 262k tokens of native context, and its hybrid Gated DeltaNet attention keeps the memory cost of long context low. It is a vision model as well, but this project uses it text-only.

In this project you will do the following:

• Install llama.cpp and the Hugging Face CLI
• Download the 4-bit quant, the 1-bit quant, and the model's MTP head
• Serve the model locally and benchmark both quants, with and without MTP speculation
• Run Claude Code against your own machine, fully offline
• Expose the model to your network through an HTTP agent harness

The raw benchmark output is in results/; every filename says which quant produced it. All commands are run from the repository root.

DISCLAIMER

Everything here was measured on a single machine with a small set of cases and reflects one moment in time; repeat runs on the same day varied by around 20% with system load. This is an independent project with no affiliation to or endorsement from Alibaba, the Qwen team, Unsloth, or Anthropic. Check the numbers on your own hardware before relying on them.

1. Install

brew install llama.cpp
brew install hf

Qwen3.8 uses the qwen3_5 architecture, supported in llama.cpp since the Qwen3.5 series. The numbers in this repository were produced on build b10355, the same build used in the Glimmer project. Verify with llama-server --version.

2. Download the model

hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q4_K_XL.gguf --local-dir models/qwen3.8
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-IQ1_S.gguf --local-dir models/qwen3.8
hf download ggml-org/Qwen3.8-27B-GGUF mtp-Qwen3.8-27B-Q4_0.gguf --local-dir models/qwen3.8

The 4-bit quant (17.6GB), the 1-bit quant (6.2GB, Unsloth Dynamic v3.0, ~77% accuracy retention according to Unsloth), and the model's MTP head (1.7GB), which drives speculative decoding. Qwen3.8 is trained with MTP (multi-token prediction), but GGUF exports carry the head as a separate file, so it is downloaded on its own.

If you only want one quant, take the 4-bit on a machine with 24GB or more and the 1-bit below that; section 5 shows what each one gives up. Verify with ls -lh models/qwen3.8/.

3. Start the server

Standard, serving the 4-bit:

llama-server -m models/qwen3.8/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --parallel 1 --cache-reuse 256 \
  --host 127.0.0.1 --port 8000 \
  -c 131072 -ngl 99 --jinja

With MTP speculative decoding:

llama-server -m models/qwen3.8/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -md models/qwen3.8/mtp-Qwen3.8-27B-Q4_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 16 -ngld 99 \
  --parallel 1 --cache-reuse 256 \
  --host 127.0.0.1 --port 8000 \
  -c 65536 -ngl 99 --jinja

For the 1-bit, swap the model filename for Qwen3.8-27B-UD-IQ1_S.gguf; it has the memory headroom to keep -c 131072 in every mode.

The two commands differ in the speculation flags and the context size.

A model normally produces one token per forward pass, and on local hardware that pass is the bottleneck. Speculative decoding cuts the number of passes: a cheap mechanism drafts several tokens ahead, the model checks the whole draft in a single pass, and the tokens that match what it would have produced anyway are accepted as a batch. Wrong guesses are thrown away, so the output is identical to running without speculation; the only question is speed.

In the Glimmer project the drafting was done by a second, unrelated model. Qwen3.8 trains its own MTP (multi-token prediction) head for the job, which GGUF exports carry as the separate 1.7GB file from step 2; -md loads it and --spec-type draft-mtp switches it on. Whether it makes each quant faster is measured in section 5.

Or use the launcher: zsh start.sh [mode] [quant]. Mode is empty for standard, mtp for speculation, agent for MTP plus thinking disabled (the fallback described in section 7), or 32k for machines with less memory. Quant is empty for the 4-bit or 1bit for the 1-bit. For example zsh start.sh mtp 1bit.

Flag	Meaning
-ngl 99	all layers on GPU
-c 131072	128k context; the model is 262k native so no YaRN or RoPE flags are needed
--jinja	enables the model's chat template, required for tool calling
--parallel 1	one slot with the full context; with the default 4 slots large prompts fail
--cache-reuse 256	reuses cached prompt prefixes, essential for agent workloads
-md ...	the MTP head file from step 2
--spec-type draft-mtp	enables speculation from the MTP head; defaults to none
-ngld 99	MTP head on GPU
--spec-draft-n-max 16	tokens drafted per round; replaces the removed --draft-max

Two memory notes. Only one layer in four is a conventional attention layer with a KV cache, so a 128k window costs about 8GB; a model with attention in every layer would need roughly four times that.

The 4-bit weights plus the MTP head plus 128k of context still exceed the GPU memory of a 36GB machine, and the server dies mid-request with Insufficient Memory. That is why the speculation modes serve 64k on the 4-bit, and start.sh handles this automatically.

The built-in browser tools enabled by start.sh, serving on your local network, and API keys work the same for any llama-server model and are covered in the Glimmer README. Keep --jinja or clients will report that the model does not support tools, and only bind 0.0.0.0 on a network you trust.

4. Verify

curl http://127.0.0.1:8000/health

The endpoint returns {"status":"ok"} when the model is loaded. Then:

curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8","messages":[{"role":"user","content":"say hi"}],"max_tokens":600}' \
  | python3 -c 'import json,sys; b=json.load(sys.stdin); m=b["choices"][0]["message"]; \
print("answer:  ", m.get("content")); \
print("thinking:", (m.get("reasoning_content") or "")[:120]); \
print("speed:   ", round(b["timings"]["predicted_per_second"],1), "tok/s")'

The output looks like this, with the speed depending on your quant and hardware:

answer:   Hi there!
thinking: 
speed:    24.6 tok/s

Qwen3.8 is a reasoning model and thinking is on by default. The reasoning is returned in reasoning_content and the answer in content, and both come out of the same token budget, so a small max_tokens produces an empty answer.

The thinking line can also come back empty: the model decides how much reasoning a request needs, and on a trivial prompt it may answer directly. An empty thinking: with a populated answer: is normal.

Thinking is controlled with the reasoning_effort request parameter; "none" disables it. In mtp mode, speculation is active only if draft_n appears in the response timings.

5. Benchmark

python3 bench/run_bench.py --target local --label baseline-4bit

Runs the cases in bench/cases.json, prints pass/fail and tok/s, and writes JSON to results/. Works with any OpenAI compatible server.

The label names the run and becomes part of the output filename. Put the quant and the mode in it (baseline-4bit, mtp-1bit, and so on) so every result file says what produced it.

The first three cases are identical to the Glimmer repo so the numbers compare directly.

Reproducing the results

Each cell of the measured performance table comes from a specific run. For each quant in turn:

1. Start the standard server (zsh start.sh for the 4-bit, zsh start.sh standard 1bit for the 1-bit), wait for /health, and run python3 bench/depth_bench.py --target local --label depth-<quant>. It sends a single ~24k-token prompt and prints the prefill and deep-context rows. Run it before anything else on a fresh server; once the prefix cache has absorbed the prompt, the prefill number means nothing.
2. On the same server, run the suite with --label baseline-<quant>. This produces the fresh-context generation row and the pass count.
3. Stop the server with Ctrl+C, start it with zsh start.sh mtp (add 1bit for the 1-bit), and run the suite with --label mtp-<quant>. Compare mean_tok_s against the baseline file; that ratio is the MTP row. The response timings show draft_n and draft_n_accepted while speculation is running.
4. The Claude Code rows are timed by hand with the server still in mtp mode; section 7 explains how.

Measured performance

All numbers from a MacBook Pro (M4 Max, 36GB) on llama.cpp b10355, measured 2026-08-23, both quants back to back under the same conditions. The raw JSON for every cell is in results/.

Scenario	Qwen3.8-27B 1-bit	Qwen3.8-27B 4-bit	Glimmer-30B, same machine
Generation, fresh context	~15 tok/s	~15 tok/s	~22 tok/s
Generation, ~24.5k tokens deep	~11 tok/s	~8 tok/s	~7 tok/s (at ~29k)
Prompt prefill	~144 tok/s	~137 tok/s	~110 tok/s
MTP speculative decoding	1.35x, 1.56x on long code	1.03x, 1.41x on long code	0.9-1.0x (dflash)
Claude Code first response (cold)	5 min 55 s	4 min 36 s	~4 min
Claude Code later turns (same session, cached)	3 min 32 s, incoherent answer	31 s	~1 min
Benchmark cases	3/4 pass	3/4 pass	n/a
"Create a deep neural network"	fails to understand the request	complete, structured implementation; one runtime bug	n/a

What the table means in practice:

The two quants generate at the same speed fresh, and the difference is capability, not speed. The 6.2GB file is not a faster ticket to the same model.

Asked to create a deep neural network, the 1-bit replied "I'm not sure what you mean there" and hallucinated a file path. In a fresh session it produced no file at all, ending its output on a leaked </think> tag. Its cached turns are not only slower than the 4-bit's, they degrade: asked what port start.sh serves, it answered I can't tell from and stopped.

The 4-bit understood the neural network request immediately and wrote a 10.7KB layered implementation with backpropagation, an XOR demo, and a regression demo. The file compiles, then hits an IndexError in the backward pass when run. A second, independent attempt under the exact test protocol produced a different implementation with the same class of bug: an IndexError in its backward pass. Backpropagation indexing appears to be precisely where this model's unaided code breaks.

Neither is frontier-class unaided, but one is a junior developer having a bad day and the other did not understand the question. The accuracy a 1-bit quant gives up is concentrated exactly where the agentic claims live.

MTP speculation is worth having on this model. It is the first speculation setup across both projects that clearly wins, 1.35x overall on the 1-bit and up to 1.56x on long code generation, against the break-even dflash results on Glimmer. Gains on the 4-bit are smaller. Short answers see no benefit either way.

The failed benchmark case is thinking, not capability. smoke-longcode fails on both quants the same way: the model spends the whole 3072-token budget on reasoning and returns no code. The fourth case is the identical prompt with reasoning_effort set to none, which disables reasoning and passes on both. When thinking is on, size max_tokens for the reasoning plus the answer.

6. Console

The server ships a browser chat UI at the address below, handy for talking to the model with nothing else in the way.

http://127.0.0.1:8000

7. Claude Code

For Claude Code, serve the model with zsh start.sh mtp (add 1bit to run the 1-bit, though the table above shows what to expect from it as an agent).

Thinking stays on, because the model's agentic reasoning is the point of running it, but it has to be bounded. Left unbounded, the model can reason for thousands of tokens before every tool call, and a single turn takes tens of minutes. The LiteLLM config caps it by sending reasoning_effort: low with every request, which is the depth control Qwen3.8 documents.

If turns still run too long for your hardware, zsh start.sh agent disables thinking entirely (--reasoning-budget 0). That makes turns fast and predictable at the cost of the reasoning the model is built around.

Claude Code speaks the Anthropic Messages API and llama-server speaks OpenAI, so LiteLLM sits between them and translates, the same as in the Glimmer project. The setup below only needs doing once. LiteLLM does not run on Python 3.14 and current FastAPI releases break it, so both pins matter:

python3.13 -m venv .venv
.venv/bin/pip install "litellm[proxy]" "fastapi<0.116"

With llama-server running:

.venv/bin/litellm --config litellm-config.yaml --port 4000 --host 127.0.0.1

Then in a new terminal, change to the repository root first; Claude Code works on whatever directory it is started from, and started anywhere else it will answer questions about the wrong files:

export ANTHROPIC_BASE_URL=http://127.0.0.1:4000
export ANTHROPIC_AUTH_TOKEN=local
export ANTHROPIC_MODEL=qwen3.8
export ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3.8
export API_TIMEOUT_MS=600000
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192
export DISABLE_NON_ESSENTIAL_MODEL_CALLS=1
claude --disallowedTools Artifact

CLAUDE_CODE_MAX_CONTEXT_TOKENS must match the served context: 65536 for the 4-bit in mtp or agent mode, 131072 for the 1-bit. The other variables point the client at LiteLLM, keep the background model local, and give the cold prefill time to finish.

The last variable and the --disallowedTools flag fix two failures that came up while building this project:

• Failed to initialize samplers: failed to parse grammar on every request. llama-server compiles the client's tool schemas into a sampling grammar. The Artifact tool's schema contains a regex with \d, the converter passes the escape through, and the grammar parser rejects it. --disallowedTools Artifact removes the one schema that trips it.
• Prefix cache eviction between turns. With --parallel 1 there is one cache slot. Claude Code's small background calls land between turns, evict the cached system prompt, and force a full re-prefill on every real turn. DISABLE_NON_ESSENTIAL_MODEL_CALLS=1 stops them.

The last command opens a normal Claude Code session, running entirely on your machine. To test it, ask something that forces a tool call, for example:

What files are in the bench directory?

The correct answer lists the files from the repository structure below, and the llama-server terminal shows the requests as they arrive. The first response is slow because the whole system prompt has to prefill; later turns in the same session reuse the cache and come back much faster. Exit with Ctrl+C twice.

To time the turns for the table in section 5, run one prompt in a fresh session and time it, then resume that session and time a follow-up. The first command pays the full system prompt prefill; the second reuses the cache:

time claude --disallowedTools Artifact -p "What files are in the bench directory of this repo?"
claude --disallowedTools Artifact -p --output-format json "say ok" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_id"])'
time claude --disallowedTools Artifact -p --resume <session-id> "And what port does start.sh use?"

8. Agent harness

agent/server.py is a Python harness that external systems connect to over HTTP. Each POST to /task runs a tool-calling agent loop against the local model, with list_files and read_file sandboxed to this repository, and returns the answer plus a trace of every tool call made.

With llama-server running:

python3 agent/server.py

It listens on 0.0.0.0:8090. Find the Mac's address with ipconfig getifaddr en0, then from any machine on the network:

curl -s http://<mac-ip>:8090/health
curl -s http://<mac-ip>:8090/task \
  -d '{"prompt": "How many benchmark cases are in bench/cases.json?"}'

Requests are served one at a time to match --parallel 1. llama-server can stay bound to 127.0.0.1; only the harness port faces the network. There is no authentication, so only run it on a network you trust. See agent/README.md for details, including PowerShell syntax for Windows.

Troubleshooting

Every error below was hit while building this project.

Symptom	Cause	Fix
command not found: pip or hf	Hugging Face CLI not installed	brew install hf
unknown model architecture: 'qwen3_5'	llama.cpp predates the Qwen3.5 series	brew upgrade llama.cpp (b10355 works)
failed to open GGUF ... No such file	model not downloaded	step 2
server exits at startup listing spec types	invalid --spec-type value	draft-mtp for this model's built-in MTP
model doesn't contain MTP layers at startup	the main GGUF does not include the MTP head	pass the separate head with -md mtp-Qwen3.8-27B-Q4_0.gguf
"--draft-max": the argument has been removed	renamed flag	use --spec-draft-n-max
speculative speed identical to baseline	speculation not enabled	add --spec-type draft-mtp; confirm draft_n in timings
Insufficient Memory errors mid-request, all requests 500	4-bit weights + MTP head + 128k context exceed GPU memory	serve 64k in speculation modes; start.sh does this
Failed to initialize samplers: failed to parse grammar	\d in a tool schema regex breaks the grammar converter (Claude Code's Artifact tool)	claude --disallowedTools Artifact
Claude Code turns run for many minutes with the server generating constantly	unbounded thinking; the model reasons for thousands of tokens per turn	check reasoning_effort: low is in litellm-config.yaml and restart LiteLLM; zsh start.sh agent disables thinking as a last resort
Claude Code invents file listings or describes the wrong project	started from a different directory, so its tools operate there	start claude from the repository root
Invalid model name passed in model=claude-...	a Claude Code feature requested a model name the proxy does not list	the wildcard entry in litellm-config.yaml covers this; restart LiteLLM if you changed the config
HEAD /api/hello 404 in the LiteLLM log	Claude Code's connectivity probe hits a route LiteLLM does not serve	harmless, ignore it
curl: (7) Failed to connect	server not running or still loading	start server; poll /health
clients report the model does not support tools	server launched without --jinja	relaunch with --jinja; start.sh includes it
empty content in response	reasoning used the whole token budget	raise max_tokens, or send "reasoning_effort": "none"
Context size has been exceeded	the default 4 slots share one context pool	add --parallel 1
ModuleNotFoundError: No module named 'proxy_server'	LiteLLM on Python 3.14, or FastAPI too new	venv on 3.13, pip install "fastapi<0.116"
Claude Code: "will retry, check your network"	client timeout during cold prefill	export API_TIMEOUT_MS=600000

Repository structure

bench/cases.json      test cases (first three shared with muse-glimmer-claude-code)
bench/targets.json    benchmark endpoints
bench/run_bench.py    benchmark runner
bench/depth_bench.py  prefill and deep-context measurement
agent/server.py       HTTP agent harness for external systems
litellm-config.yaml   LiteLLM proxy config
start.sh              server launcher (modes: standard / mtp / agent / 32k, quants: 4-bit / 1bit)
results/              benchmark output (JSON), quant in every filename
assets/               banner and result charts

Conclusion

The question this repository set out to answer was whether the 1-bit quant delivers what the posts claim: a frontier-class model on an ordinary computer. It does not.

The 1-bit runs, chats, and answers simple lookups. But asked to create a deep neural network it did not understand the request in conversation, produced nothing in a fresh session, and its multi-turn answers degrade into fragments. If a model fits in 8GB and cannot code, it is not a smaller frontier model; it is a different, lesser thing wearing the same name.

The 4-bit is the real article. It understood every request, worked as a Claude Code agent with 31-second cached turns, and wrote a complete, structured neural network implementation, though with a runtime bug and at roughly ten times the wall time Glimmer-30B needed for the same test.

Against Glimmer the trade is clear. Glimmer generates faster fresh and finishes tasks sooner. Qwen3.8 is faster where agents actually operate, deep in context, holds a 262k window against 32k, and is the first model in these projects whose speculative decoding actually pays. Its weakness is deliberation: even capped, it spends more time thinking about work than Glimmer spends doing it.

The practical guidance is short. On a machine with 24GB or more, run the 4-bit and cap the reasoning. On less, run the 1-bit for chat and lookups and expect nothing agentic from it.

And test the claims yourself before repeating them. Every number here took one evening and one laptop to produce.

Contributing

Issues and pull requests are welcome, especially benchmark runs on other Apple Silicon machines and new troubleshooting entries.

License

Apache License 2.0. See LICENSE.

Author