Repository navigation
Description
This report has been corrected. It originally claimed Ollama was discarding a valid tool call. That was wrong, and the details are in the comments. What follows is the corrected version. The remaining issue is that when a tool call fails to parse, the generated output is discarded with no way for a caller to see what happened.
What happens
With qwen2.5:7b-instruct on Ollama 0.32.1, a request carrying a single tool comes back as an empty assistant message: no content, no tool_calls, finish_reason "stop". The response reports 40 completion tokens, so it says it generated 40 tokens and returns none of them. There is no error, no warning, and nothing in the response that distinguishes this from a model that simply chose to say nothing.
What the model actually generated
Not visible from the API. I built v0.32.1 with #17284 applied, which returns the buffered output as content when no tool call is parsed, and re-ran the same request. The recovered output, completion_tokens 40, identical to the unpatched run:
<tool_call>
{"name": "Apply tags to a document", "arguments": {"document_id": "doc-17", "tags": ["alpha", "beta", "gamma"]}}
</tool_call>
The block is well-formed and the arguments are correct. The name is "Apply tags to a document", which is the tool's description, not its name tag_document. No registered tool matches, so the parser rejects it.
The parser is doing the right thing. The model emitted the wrong field. The defect is that this is invisible: 40 tokens of near-correct, obviously-diagnosable output are dropped and the caller is given an empty string.
Why the same weights work under llama.cpp
The original report leaned on a llama.cpp comparison: the same GGUF blob, served by llama-server --jinja, returns a correct tool call 12/12. I attributed that to Ollama's parser being at fault. That was the wrong conclusion.
llama.cpp constrains generation. It builds a GBNF grammar from the tool definitions and applies it lazily once the tool-call start token appears (build_grammar over foreach_function(inputs.tools, ...) in common/chat.cpp, with grammar_lazy set when tools are present and tool_choice is auto). The function name is constrained to the declared tools, so this failure cannot occur there. Ollama generates unconstrained and parses afterwards, so a model that puts the wrong string in name fails, and currently fails silently.
Reproduction
{
"model": "qwen2.5:7b-instruct",
"messages": [
{
"role": "user",
"content": "Tag document doc-17 with exactly alpha, beta, and gamma in that order."
}
],
"tools": [
{
"type": "function",
"function": {
"name": "tag_document",
"description": "Apply tags to a document.",
"parameters": {
"type": "object",
"required": ["document_id", "tags"],
"properties": {
"document_id": { "type": "string" },
"tags": { "type": "array", "items": { "type": "string" } }
}
}
}
}
],
"tool_choice": "auto",
"stream": false,
"temperature": 0.0,
"top_p": 1.0,
"seed": 42,
"max_tokens": 1024
}Response:
{"id":"chatcmpl-392","object":"chat.completion","created":1784508668,"model":"qwen2.5:7b-instruct","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":""},"finish_reason":"stop"}],"usage":{"prompt_tokens":229,"completion_tokens":40,"total_tokens":269}}Deterministic across reps. /api/chat behaves the same, so this is not specific to the OpenAI-compatible layer. Streaming returns a single chunk with content: "" and finish_reason stop. A second tool description produced the same result.
Environment
- Ollama 0.32.1, macOS 26.5.2, Apple Silicon (M4 Max).
- llama-server version 10050 (b15ca938a) for the comparison.
- Model
qwen2.5:7b-instruct, blobsha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730, 4683073952 bytes, Q4_K_M. The same file for both servers. - Patched build for the recovery: tag
v0.32.1(30c3903) with tools: return buffered output as content when no tool call is parsed #17284 applied cleanly,go test -run TestContent ./tools/passing.
What I think is worth fixing
- Do not discard generated output when no tool call parses. tools: return buffered output as content when no tool call is parsed #17284 does exactly this and it turned an opaque failure into a one-request diagnosis.
- The token accounting is still odd on its own terms: 40 completion tokens reported with zero bytes returned. Even with the output restored, an empty response with a nonzero
eval_countseems worth surfacing rather than reporting as a clean completion.
Related: #12174 reports lost tool calls on qwen2.5-coder with an array-typed parameter and settled on the model being unreliable. Given what turned up here, some of those reports may be the same wrong-name failure, invisible for the same reason. #16932 shares the symptom but has a different cause in the ministral parser.
Happy to test a patch against this repro, or to run it across other models if that would help.
Activity
Record ollama/ollama#17274 filing outcome; retract tool_choice defect…
rick-github commentedon Jul 20, 2026
Can you provide the content of the API call?
GuiBarradas commentedon Jul 21, 2026
On the parsing side: when the parser sees
<tool_call>but never completes a tool call (unknown name, malformed or truncated JSON), everything after the tag is silently discarded at end of stream. That matches the symptom here: tokens counted, but no content and no tool_calls.The exact output quoted above parses fine, so the model likely emitted something different through
/v1/chat/completions. I opened #17284 to return unparsed output as content instead of dropping it, which would make failures like this visible and diagnosable.devYRPauli commentedon Jul 21, 2026
Sorry, I should have included that up front. I have edited the issue body to add the exact request JSON and the verbatim response.
Short version: single user message, one tool
tag_documentwith a required stringdocument_idand a required array-of-stringtags,tool_choice: "auto",stream: false,temperature 0,top_p 1,seed 42,max_tokens 1024. The response comes back200with"content": "", notool_calls,finish_reason"stop",prompt_tokens177,completion_tokens40.Happy to provide anything else that would help.
devYRPauli commentedon Jul 21, 2026
You are right to push on this, and having tried to close it properly I have to walk part of my report back.
My "the model emits it correctly" evidence came from
/api/generatewithraw: trueand a prompt I rendered myself. That does not establish what the model emits through/v1/chat/completions, which is your point exactly. I went back to try to capture the real thing and could not.What I tried, on 0.32.1:
OLLAMA_DEBUG=1andOLLAMA_DEBUG_LOG_REQUESTS=1, with llama-server at--log-verbosity 4. Neither logs the rendered prompt or the generated text. All I get is metadata:prompt_len=790chars,n_tokens=177, and40 tokensgenerated.--log-prompts-dirwould capture it, but it is CLI-only with no env equivalent, and Ollama's internal invocation does not expose it.So I reconstructed instead. The debug log shows
selected=go_template, so I took Ollama's own stored template fromollama show qwen2.5:7b-instruct --template(plus--system), rendered it, and fed it back through/api/generatewithraw: true, seed 42, temperature 0. Two variants, with and without the default system message, both produced:That is valid and complete. But I do not want to present it as proof, because the reconstruction is not byte-exact and one number points the other way:
/v1requestThe real path generated 40 tokens; my reconstructions generate 37. Whatever the real path produced, it was not the same 37 tokens I am showing, so I cannot rule out that it emitted something malformed, truncated, or otherwise unparseable, which is precisely your hypothesis. Three tokens is about the size of the difference you would expect from a stray preamble or an unclosed block.
Where that leaves me: the parse-side symptom in the issue is solid and reproducible, but my claim about what the model generates on the failing path is not established, and I will correct the issue body to say so rather than leave an overstated claim standing.
It also seems like a decent argument for #17284. The reason I cannot settle this is that the generated output is discarded with nothing recoverable from the server side. If the buffered output came back as content, this would have been a two-minute question to answer instead of an open one.
If it would help, I am happy to run a patched build with prompt or generation logging against this repro and post the exact bytes. The model is pulled and the repro is deterministic, so it is cheap for me to rerun.
[-]qwen2.5 tool call is dropped: empty content, no tool_calls, but 40 completion tokens reported[/-][+]Tool-call output is silently discarded when the parse fails: empty content, no tool_calls, 40 completion tokens reported[/+]devYRPauli commentedon Jul 21, 2026
I built 0.32.1 with #17284 applied and ran the failing request against it. That recovers the discarded bytes, so this is now settled, and it settles against my original claim.
The 40 tokens Ollama was throwing away:
completion_tokens40, identical to the unpatched run, so this is the real generation and not a reconstruction. A same-session control against the stock server returnedcontent: ""as before.@GuiBarradas was right. The block is well-formed and the arguments are correct, but
nameis"Apply tags to a document", which is the tool's description, not its nametag_document. No tool matches that name, so the parser rejects it. The parser is behaving correctly and I was wrong to call this a dropped valid tool call. I have corrected the issue body and title.Build details for anyone reproducing: tag
v0.32.1(30c3903), #17284 applied cleanly to it, both hunks,go test -run TestContent ./tools/passing.What I think is still worth something here:
The reason the same GGUF returns a correct call under llama.cpp 12/12 is not that its parser is more forgiving. llama.cpp builds a GBNF grammar from the tool definitions and applies it lazily once the tool-call start token appears (
build_grammaroverforeach_function(inputs.tools, ...)incommon/chat.cpp, withgrammar_lazyset when tools are present andtool_choiceis auto). The function name is constrained to the declared tools, so this particular failure cannot occur there. Ollama generates unconstrained and parses afterwards, so a model that picks the wrong field fornamefails, and currently fails invisibly.That makes this a fairly clean argument for #17284. With the patch, one request shows exactly what went wrong. Without it, the API reports 40 tokens generated, returns zero bytes, and gives the caller nothing to go on. I spent a while on the wrong theory precisely because that information is unavailable, and I would guess a share of the other "tools just do not work with model X" reports are the same mistake.
If it would be useful I am happy to test #17284 against other models and post what turns up, or to check whether grammar-constrained tool calling has been considered separately.
Apologies for the noise on the original diagnosis.
GuiBarradas commentedon Jul 22, 2026
@devYRPauli Thanks for taking it all the way to a patched build, that settles it better than any reconstruction could. On the offer to test other models: for #17284 the interesting cases are text-tag templates (qwen2.5, mistral style), anything where a failed parse currently returns an empty message. The bare-JSON side of the same failure mode turned out to be a separate bug, filed as #17323 with a fix in #17324, if you want another one to poke at. And the lazy grammar point is a good one, probably worth its own feature request so it does not get buried in this thread.
devYRPauli commentedon Aug 24, 2026
Sorry for the slow reply. I ran the text-tag template cases you asked about.
Result: the drop still happens on ollama 0.32.15, and it is not limited to the OpenAI-compatible route.
Models tested, both text-tag template (
<tool_call>style):qwen2.5:0.5b""qwen3:0.6b""Both endpoints drop it. Same request, same seed:
So the loss is in the tool-call parsing path that both routes share, not in the
/v1translation. My original report only showed/v1, so this widens it.Control, same prompt and seed with no
toolsfield:The model generates normal output for this prompt. With
toolsregistered it generates 40 tokens and the caller gets an empty message. The tokens are real generation, not an empty completion.How I triggered it: one tool
tag_document, descriptionApply tags to a document, and a prompt that pushes the model to use the description as the function name. That reproduces the exact payload from my first report, where the emitted name wasApply tags to a documentand did not match any registered tool. It is a forced case, not an organic one, but it is the same failure condition.Cases that did NOT reproduce, for completeness:
tool_calls<tool_call>: inconclusive. The model emitted awell-formed call instead of echoing the malformed text, so this never reached the parser.
On this evidence the trigger is a name that does not match a registered tool. I did not
manage to test malformed JSON, so I cannot say whether that path drops output too.
One limit on this: I did not build with #17284 applied, because I have no Go toolchain on this machine. These numbers measure the failure on stock 0.32.15. They do not verify your fix. If a build with #17284 returns those 40 tokens as content, that closes the case for this model family.
aldi949 commentedon Sep 11, 2026
Hey @devYRPauli — I found #17274 while going through tool-calling failures where the API response hides most of what actually went wrong.
The “tokens were generated, but the caller gets empty content and no tool_calls” behavior is very close to the class of problems I’ve been working on.
I’m building ToolCall Doctor:
https://github.com/aldi949/toolcall-doctor
The idea is to take a failing tool-calling request and experimentally narrow down where the failure comes from and, when possible, verify a workaround.
I recently froze
v0.3.0-rc2, and I’m looking for external cases to run against it without modifying Doctor for the case first.If you can still reproduce #17274, would you be interested in running the frozen version in your existing Ollama setup and seeing what it finds?
If yes, I can send you a minimal set of steps.
devYRPauli commentedon Sep 14, 2026
Still reproduces on 0.34.0, the current release. Same request as in the issue body, run against
qwen2.5:0.5b:/v1/chat/completions""/api/chat""/api/chat, notoolsfieldThat is three releases: 0.32.1 in the first report, 0.32.15 in August, 0.34.0 today.
I also re-checked #17284 against
mainat53fed26. It applies cleanly, andgo test ./tools/returns rc 0 both before and after it.Then I fed the exact bytes from my first report through the parser on
53fed26, withtag_documentregistered, one character at a time:mainat53fed26<tool_call>blockSo the fix still does what it says on current main, and the failure it addresses is still live on the current release.
@aldi949 the case is already reduced to a single request. The exact JSON is in the issue body and it reproduces on a 0.5b model. You can run Doctor against it directly, without my setup. The result it should reach is above: the model puts the tool description in
name, no registered tool matches, and the buffered output is dropped.rick-github commentedon Sep 14, 2026
A simple fix is just to have the model use the embedded Jinja template, which will then result in the runner using constrained generation.
ollama create qwen2.5:7b-instruct-jinja -f <(ollama show --modelfile qwen2.5:7b-instruct | egrep "^(FROM|SYSTEM|PARAMETER)")If using modified models is not acceptable,
OLLAMA_GO_TEMPLATE=0can be set in the server environment and all models will use their embedded Jinja template if a PARSER is not configured.$ OLLAMA_GO_TEMPLATE=0 ollama servedevYRPauli commentedon Sep 14, 2026
Confirmed here on
qwen2.5:0.5b. Both of your routes produce the tool call.qwen2.5:0.5bqwen2.5:0.5b-jinjaOLLAMA_GO_TEMPLATE=0qwen2.5:0.5bI built the jinja variant with your exact recipe.
That is a working route today, and it is the answer I would give someone hitting this in production.
It does not cover what the issue reports. Row 1 generated 40 tokens and returned 0 bytes. The caller sees
content: "",done_reason: stop, and no error. Nothing tells it that output was produced and then dropped.Constrained generation stops this model from putting an unmatched name in that field. It does not change what the Go template path does when a name still does not match. That is the part #17284 changes: return the buffered text as
contentinstead of discarding it.Any model on the Go template path can still emit a name that matches no registered tool. When it does, the bytes go nowhere.
rick-github commentedon Sep 14, 2026
So don't use the Go template. I'm not privy to the roadmap that the maintainers have, but the CGO engine has been removed. Moving forward, ollama is likely to transition into a model manager/scheduler for the various runners - llama-server, MLX. No models added to the library since the switch to 0.30.0 include a Go template, so it seems that legacy support will continue but the road ahead is either PARSER or Jinja.
devYRPauli commentedon Sep 14, 2026
Agreed on direction. Nothing in #17284 adds to the Go path or slows the move to PARSER and Jinja.
Row 1 of my table is a plain
ollama pull qwen2.5:0.5b. No template I wrote, no flag. That model is in the library today and it takes the Go path. The caller did not choose that template.What the path does now is the part I want to fix. It buffers the output, finds no tool with that name, and drops the bytes. The caller sees
content: "",done_reason: stop, and no error. Nothing in the response says text was produced.#17284 changes 5 lines in
tools/tools.go. It returns the buffered text ascontentinstead of discarding it. Legacy support that silently loses output is worse than legacy support that returns it.rick-github commentedon Sep 14, 2026
Or, set
OLLAMA_GO_TEMPLATE=0and get an actual tool call, not text that the client needs to interpret.devYRPauli commentedon Sep 14, 2026
OLLAMA_GO_TEMPLATE=0does give a tool call. I measured it again on 0.34.0 just now, and it is row 3 of the table above.You are right that #17284 returns raw text. The content it returns is the
<tool_call>block, and the client has to read it.The part I am arguing is what happens when nobody sets that variable.
envconfig.Varreads it from the server process environment. There is no per-request or per-model override in the API. A caller who does not run the server cannot set it.The server already makes this choice on its own.
shouldPreferChatTemplateinserver/images.goswitches to the GGUF chat template when the Go template cannot round-trip tools. Forqwen2.5:0.5bboth templates report[completion tools], and the Go template passesgoTemplateHasToolRoundTrip, so the server keeps the Go one.Two runs on 0.34.0, same request, same seed, from the server's own
template selectionlog line:go_templateOLLAMA_GO_TEMPLATE=0gguf_chat_templateRow 1 is what this model does today with no environment change. It generates 40 tokens, returns zero bytes, and reports
done_reason: stopwith no error.aldi949 commentedon Sep 14, 2026
@devYRPauli Thanks — I ran the exact issue-body request against Ollama 0.34.0 + qwen2.5:0.5b.
I reproduced the silent-drop behavior on
/api/chat2/3 times, but not stably 3/3. On/v1/chat/completionsI got 0/3 reproductions of the target failure class in my environment.Before I adapt ToolCall Doctor to support this failure predicate, could you share the exact Ollama launch command, model state/config, and request command you used for your stable reproduction?
I want to match your environment exactly rather than accidentally validate a nearby failure variant.
devYRPauli commentedon Sep 14, 2026
Here is the exact setup.
Server:
ollama servewith no environment variables set. Version 0.34.0, fromcurl -s http://127.0.0.1:11434/api/version.Model:
qwen2.5:0.5b, IDa8b0c5157701, blobsha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515, Q4_K_M, 494.03M parameters. The issue body namesqwen2.5:7b-instruct. My table above moved to0.5b, so use0.5bto match my numbers.Your 2/3 on
/api/chathas a cause./api/chatdoes not readtemperature,top_porseedat the top level.ChatRequestinapi/types.godeclares no such fields. Sampling parameters reach the runner only throughOptions. A top-level"seed": 42is dropped by the JSON decode with no error, so the run is unseeded.Five runs of each shape, same server, same model, nothing else changed:
optionsThe top-level shape is unseeded and drops the call 3 times in 5. The
optionsshape is seeded and drops it 5 times in 5. Use the second one as the predicate.The failure is
message.tool_callsabsent,message.contentempty,eval_count40,done_reasonstop./v1/chat/completionsreproduces 5 times in 5 here, so I cannot match your 0/3.openai/openai.gocopies top-leveltemperature,seedandtop_pintooptions, so the request shape is not the difference. I ran the issue-body JSON with the model changed toqwen2.5:0.5band gottool_calls0, content 0 bytes,completion_tokens40,finish_reasonstop, on all five.One line in the server log tells you whether you are on the same path:
If yours says
selected=gguf_chat_template, the server picked the other template and you get a parsed tool call. That is row 2 of my table above, and it is not the failure you want to target.aldi949 commentedon Sep 14, 2026
@devYRPauli Thanks — I matched the setup more closely and got a different result.
Verified:
ollama serveselected=go_templateoptionsBut both routes now produce a valid
tag_documenttool call 5/5:/api/chat/v1/chat/completionsSo I still cannot reproduce your 40-token silent-drop state.
Could you share the exact output of:
ollama show qwen2.5:0.5b --modelfileand, if possible, the exact Ollama build/commit info plus OS/architecture?
I want to diff the model/template metadata and runner environment against mine, because the request path is now confirmed identical.
devYRPauli commentedon Sep 14, 2026
Reproduced again on the same box, 5/5, with your setup: 0.34.0,
qwen2.5:0.5b, plainollama serve, no env overrides,selected=go_template, sampling insideoptions.Here is the measurement that separates our runs. The tokens are generated and then dropped.
/api/chatwithtools""/api/chat, same schema pasted into a system message, notoolskeySame endpoint, same model, same options, same prompt length. Without the
toolskey the server returns this ascontent:With the
toolskey it returnscontent: "",tool_calls: null,done_reason: "stop",eval_count: 40. Streamed, that request produces exactly one chunk, the final done chunk, with empty content. So the runner generates 40 tokens and none of them reach the caller.The drop does not depend on the schema. Four variants, all with
toolspresent:itemsdocument_iditemsremovedeval_countmoves with the prompt, so the model generates real output every time.Your 37 is my no-parser number, which points at the backend. Mine:
macOS 26.6.2 (build 25G83), Darwin 25.6.0, arm64, Apple M1 Pro, 16 GiB, ollama 0.34.0 installed from Homebrew. What does the
inference computeline say on yours?A 0.5B Q4_K_M model at greedy decoding can flip an argmax between backends, and that changes the continuation. That accounts for 37 against 40 on the same prompt.
The backend only decides who lands on it. The defect is what happens next. When the output does not match the parser, ollama returns an empty assistant message with
done_reason: "stop"and no error. A caller cannot tell that apart from a model that chose to say nothing, and the 40 tokens are gone.ollama show qwen2.5:0.5b --modelfilegives the stock qwen2.5 chat template,SYSTEM You are Qwen, created by Alibaba Cloud. You are a helpful assistant., and noPARAMETERlines at all. The blob issha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515, Q4_K_M, 494.03M. Worth diffing that blob digest against yours.aldi949 commentedon Sep 14, 2026
@devYRPauli Thanks — that isolates the remaining difference very clearly. My local reproduction is CPU, while yours is Metal, and everything else now matches closely.
Since you already have the target failure stable 5/5, would you be willing to run ToolCall Doctor against the reproducer on your machine? I can give you the exact command/setup so it should only take a few minutes.
That would let us test Doctor against the real failure path instead of trying to reproduce Metal behavior on a different backend.
devYRPauli commentedon Sep 14, 2026
Doctor can be tested against this without my machine. The drop is in the parser, and it reproduces with no model, no server and no backend.
tools/tools.go:394onmainat53fed26:p.ncounts parsed tool calls. When zero calls parse, a{or[tag hands the buffer back and a string tag returns an empty string.I fed the recovered block from the issue body to
NewParserWithTagonce per tag, with the tool registered astag_document:Content()main53fed26<tool_call>main53fed26{<tool_call>{The buffer holds 137 bytes in every row. Only the string-tag row on
mainthrows them away.#17284 deletes that special case, and the whole
toolspackage passes on its head. GitHub reports it mergeable and 1 commit behindmain.Metal only decides whether the model emits a name that matches no registered tool. After that the loss is deterministic and the backend does not matter. So the predicate Doctor can target is: the tag was found, zero calls parsed, and the tag is not
{or[.