Skip to content

Tool-call output is silently discarded when the parse fails: empty content, no tool_calls, 40 completion tokens reported #17274

Description

@devYRPauli

This report has been corrected. It originally claimed Ollama was discarding a valid tool call. That was wrong, and the details are in the comments. What follows is the corrected version. The remaining issue is that when a tool call fails to parse, the generated output is discarded with no way for a caller to see what happened.

What happens

With qwen2.5:7b-instruct on Ollama 0.32.1, a request carrying a single tool comes back as an empty assistant message: no content, no tool_calls, finish_reason "stop". The response reports 40 completion tokens, so it says it generated 40 tokens and returns none of them. There is no error, no warning, and nothing in the response that distinguishes this from a model that simply chose to say nothing.

What the model actually generated

Not visible from the API. I built v0.32.1 with #17284 applied, which returns the buffered output as content when no tool call is parsed, and re-ran the same request. The recovered output, completion_tokens 40, identical to the unpatched run:

<tool_call>
{"name": "Apply tags to a document", "arguments": {"document_id": "doc-17", "tags": ["alpha", "beta", "gamma"]}}
</tool_call>

The block is well-formed and the arguments are correct. The name is "Apply tags to a document", which is the tool's description, not its name tag_document. No registered tool matches, so the parser rejects it.

The parser is doing the right thing. The model emitted the wrong field. The defect is that this is invisible: 40 tokens of near-correct, obviously-diagnosable output are dropped and the caller is given an empty string.

Why the same weights work under llama.cpp

The original report leaned on a llama.cpp comparison: the same GGUF blob, served by llama-server --jinja, returns a correct tool call 12/12. I attributed that to Ollama's parser being at fault. That was the wrong conclusion.

llama.cpp constrains generation. It builds a GBNF grammar from the tool definitions and applies it lazily once the tool-call start token appears (build_grammar over foreach_function(inputs.tools, ...) in common/chat.cpp, with grammar_lazy set when tools are present and tool_choice is auto). The function name is constrained to the declared tools, so this failure cannot occur there. Ollama generates unconstrained and parses afterwards, so a model that puts the wrong string in name fails, and currently fails silently.

Reproduction

{
  "model": "qwen2.5:7b-instruct",
  "messages": [
    {
      "role": "user",
      "content": "Tag document doc-17 with exactly alpha, beta, and gamma in that order."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "tag_document",
        "description": "Apply tags to a document.",
        "parameters": {
          "type": "object",
          "required": ["document_id", "tags"],
          "properties": {
            "document_id": { "type": "string" },
            "tags": { "type": "array", "items": { "type": "string" } }
          }
        }
      }
    }
  ],
  "tool_choice": "auto",
  "stream": false,
  "temperature": 0.0,
  "top_p": 1.0,
  "seed": 42,
  "max_tokens": 1024
}

Response:

{"id":"chatcmpl-392","object":"chat.completion","created":1784508668,"model":"qwen2.5:7b-instruct","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":""},"finish_reason":"stop"}],"usage":{"prompt_tokens":229,"completion_tokens":40,"total_tokens":269}}

Deterministic across reps. /api/chat behaves the same, so this is not specific to the OpenAI-compatible layer. Streaming returns a single chunk with content: "" and finish_reason stop. A second tool description produced the same result.

Environment

  • Ollama 0.32.1, macOS 26.5.2, Apple Silicon (M4 Max).
  • llama-server version 10050 (b15ca938a) for the comparison.
  • Model qwen2.5:7b-instruct, blob sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730, 4683073952 bytes, Q4_K_M. The same file for both servers.
  • Patched build for the recovery: tag v0.32.1 (30c3903) with tools: return buffered output as content when no tool call is parsed #17284 applied cleanly, go test -run TestContent ./tools/ passing.

What I think is worth fixing

  1. Do not discard generated output when no tool call parses. tools: return buffered output as content when no tool call is parsed #17284 does exactly this and it turned an opaque failure into a one-request diagnosis.
  2. The token accounting is still odd on its own terms: 40 completion tokens reported with zero bytes returned. Even with the output restored, an empty response with a nonzero eval_count seems worth surfacing rather than reporting as a clean completion.

Related: #12174 reports lost tool calls on qwen2.5-coder with an array-typed parameter and settled on the model being unreliable. Given what turned up here, some of those reports may be the same wrong-name failure, invisible for the same reason. #16932 shares the symptom but has a different cause in the ministral parser.

Happy to test a patch against this repro, or to run it across other models if that would help.

Activity

  1. added a commit that references this issue on Jul 20, 2026
  2. rick-github commented on Jul 20, 2026

    @rick-github
    Collaborator

    Reproduction

    Can you provide the content of the API call?

  3. GuiBarradas commented on Jul 21, 2026

    @GuiBarradas

    On the parsing side: when the parser sees <tool_call> but never completes a tool call (unknown name, malformed or truncated JSON), everything after the tag is silently discarded at end of stream. That matches the symptom here: tokens counted, but no content and no tool_calls.

    The exact output quoted above parses fine, so the model likely emitted something different through /v1/chat/completions. I opened #17284 to return unparsed output as content instead of dropping it, which would make failures like this visible and diagnosable.

  4. devYRPauli commented on Jul 21, 2026

    @devYRPauli
    Author

    Sorry, I should have included that up front. I have edited the issue body to add the exact request JSON and the verbatim response.

    Short version: single user message, one tool tag_document with a required string document_id and a required array-of-string tags, tool_choice: "auto", stream: false, temperature 0, top_p 1, seed 42, max_tokens 1024. The response comes back 200 with "content": "", no tool_calls, finish_reason "stop", prompt_tokens 177, completion_tokens 40.

    Happy to provide anything else that would help.

  5. devYRPauli commented on Jul 21, 2026

    @devYRPauli
    Author

    You are right to push on this, and having tried to close it properly I have to walk part of my report back.

    My "the model emits it correctly" evidence came from /api/generate with raw: true and a prompt I rendered myself. That does not establish what the model emits through /v1/chat/completions, which is your point exactly. I went back to try to capture the real thing and could not.

    What I tried, on 0.32.1:

    • OLLAMA_DEBUG=1 and OLLAMA_DEBUG_LOG_REQUESTS=1, with llama-server at --log-verbosity 4. Neither logs the rendered prompt or the generated text. All I get is metadata: prompt_len=790 chars, n_tokens=177, and 40 tokens generated.
    • llama-server's --log-prompts-dir would capture it, but it is CLI-only with no env equivalent, and Ollama's internal invocation does not expose it.

    So I reconstructed instead. The debug log shows selected=go_template, so I took Ollama's own stored template from ollama show qwen2.5:7b-instruct --template (plus --system), rendered it, and fed it back through /api/generate with raw: true, seed 42, temperature 0. Two variants, with and without the default system message, both produced:

    <tool_call>
    {"name": "tag_document", "arguments": {"document_id": "doc-17", "tags": ["alpha", "beta", "gamma"]}}
    </tool_call>
    

    That is valid and complete. But I do not want to present it as proof, because the reconstruction is not byte-exact and one number points the other way:

    prompt tokens completion tokens
    real /v1 request 177 40
    reconstruction A 183 37
    reconstruction B 167 37

    The real path generated 40 tokens; my reconstructions generate 37. Whatever the real path produced, it was not the same 37 tokens I am showing, so I cannot rule out that it emitted something malformed, truncated, or otherwise unparseable, which is precisely your hypothesis. Three tokens is about the size of the difference you would expect from a stray preamble or an unclosed block.

    Where that leaves me: the parse-side symptom in the issue is solid and reproducible, but my claim about what the model generates on the failing path is not established, and I will correct the issue body to say so rather than leave an overstated claim standing.

    It also seems like a decent argument for #17284. The reason I cannot settle this is that the generated output is discarded with nothing recoverable from the server side. If the buffered output came back as content, this would have been a two-minute question to answer instead of an open one.

    If it would help, I am happy to run a patched build with prompt or generation logging against this repro and post the exact bytes. The model is pulled and the repro is deterministic, so it is cheap for me to rerun.

  6. changed the title [-]qwen2.5 tool call is dropped: empty content, no tool_calls, but 40 completion tokens reported[/-] [+]Tool-call output is silently discarded when the parse fails: empty content, no tool_calls, 40 completion tokens reported[/+] on Jul 21, 2026
  7. devYRPauli commented on Jul 21, 2026

    @devYRPauli
    Author

    I built 0.32.1 with #17284 applied and ran the failing request against it. That recovers the discarded bytes, so this is now settled, and it settles against my original claim.

    The 40 tokens Ollama was throwing away:

    <tool_call>
    {"name": "Apply tags to a document", "arguments": {"document_id": "doc-17", "tags": ["alpha", "beta", "gamma"]}}
    </tool_call>
    

    completion_tokens 40, identical to the unpatched run, so this is the real generation and not a reconstruction. A same-session control against the stock server returned content: "" as before.

    @GuiBarradas was right. The block is well-formed and the arguments are correct, but name is "Apply tags to a document", which is the tool's description, not its name tag_document. No tool matches that name, so the parser rejects it. The parser is behaving correctly and I was wrong to call this a dropped valid tool call. I have corrected the issue body and title.

    Build details for anyone reproducing: tag v0.32.1 (30c3903), #17284 applied cleanly to it, both hunks, go test -run TestContent ./tools/ passing.

    What I think is still worth something here:

    The reason the same GGUF returns a correct call under llama.cpp 12/12 is not that its parser is more forgiving. llama.cpp builds a GBNF grammar from the tool definitions and applies it lazily once the tool-call start token appears (build_grammar over foreach_function(inputs.tools, ...) in common/chat.cpp, with grammar_lazy set when tools are present and tool_choice is auto). The function name is constrained to the declared tools, so this particular failure cannot occur there. Ollama generates unconstrained and parses afterwards, so a model that picks the wrong field for name fails, and currently fails invisibly.

    That makes this a fairly clean argument for #17284. With the patch, one request shows exactly what went wrong. Without it, the API reports 40 tokens generated, returns zero bytes, and gives the caller nothing to go on. I spent a while on the wrong theory precisely because that information is unavailable, and I would guess a share of the other "tools just do not work with model X" reports are the same mistake.

    If it would be useful I am happy to test #17284 against other models and post what turns up, or to check whether grammar-constrained tool calling has been considered separately.

    Apologies for the noise on the original diagnosis.

  8. GuiBarradas commented on Jul 22, 2026

    @GuiBarradas

    @devYRPauli Thanks for taking it all the way to a patched build, that settles it better than any reconstruction could. On the offer to test other models: for #17284 the interesting cases are text-tag templates (qwen2.5, mistral style), anything where a failed parse currently returns an empty message. The bare-JSON side of the same failure mode turned out to be a separate bug, filed as #17323 with a fix in #17324, if you want another one to poke at. And the lazy grammar point is a good one, probably worth its own feature request so it does not get buried in this thread.

  9. devYRPauli commented on Aug 24, 2026

    @devYRPauli
    Author

    Sorry for the slow reply. I ran the text-tag template cases you asked about.

    Result: the drop still happens on ollama 0.32.15, and it is not limited to the OpenAI-compatible route.

    Models tested, both text-tag template (<tool_call> style):

    model completion tokens generated content returned tool_calls
    qwen2.5:0.5b 40 "" none
    qwen3:0.6b 165 "" none

    Both endpoints drop it. Same request, same seed:

    /v1/chat/completions   content=''  tool_calls=None  finish_reason=stop  completion_tokens=40
    /api/chat              content=''  tool_calls=None  done_reason=stop    eval_count=40
    

    So the loss is in the tool-call parsing path that both routes share, not in the /v1 translation. My original report only showed /v1, so this widens it.

    Control, same prompt and seed with no tools field:

    /api/chat  content='Certainly! To apply tags to a document, I will need the actual text ...'  eval_count=45
    

    The model generates normal output for this prompt. With tools registered it generates 40 tokens and the caller gets an empty message. The tokens are real generation, not an empty completion.

    How I triggered it: one tool tag_document, description Apply tags to a document, and a prompt that pushes the model to use the description as the function name. That reproduces the exact payload from my first report, where the emitted name was Apply tags to a document and did not match any registered tool. It is a forced case, not an organic one, but it is the same failure condition.

    Cases that did NOT reproduce, for completeness:

    • normal tool use: parsed fine, returned a correct tool_calls
    • a request naming actions with no registered tool: returned prose, no drop
    • an attempt to force truncated JSON inside <tool_call>: inconclusive. The model emitted a
      well-formed call instead of echoing the malformed text, so this never reached the parser.

    On this evidence the trigger is a name that does not match a registered tool. I did not
    manage to test malformed JSON, so I cannot say whether that path drops output too.

    One limit on this: I did not build with #17284 applied, because I have no Go toolchain on this machine. These numbers measure the failure on stock 0.32.15. They do not verify your fix. If a build with #17284 returns those 40 tokens as content, that closes the case for this model family.

  10. aldi949 commented on Sep 11, 2026

    @aldi949

    Hey @devYRPauli — I found #17274 while going through tool-calling failures where the API response hides most of what actually went wrong.
    The “tokens were generated, but the caller gets empty content and no tool_calls” behavior is very close to the class of problems I’ve been working on.
    I’m building ToolCall Doctor:
    https://github.com/aldi949/toolcall-doctor
    The idea is to take a failing tool-calling request and experimentally narrow down where the failure comes from and, when possible, verify a workaround.
    I recently froze v0.3.0-rc2, and I’m looking for external cases to run against it without modifying Doctor for the case first.
    If you can still reproduce #17274, would you be interested in running the frozen version in your existing Ollama setup and seeing what it finds?
    If yes, I can send you a minimal set of steps.

  11. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    Still reproduces on 0.34.0, the current release. Same request as in the issue body, run against qwen2.5:0.5b:

    route content tool_calls tokens generated
    /v1/chat/completions "" none 40
    /api/chat "" none 40
    /api/chat, no tools field prose none 13

    That is three releases: 0.32.1 in the first report, 0.32.15 in August, 0.34.0 today.

    I also re-checked #17284 against main at 53fed26. It applies cleanly, and go test ./tools/ returns rc 0 both before and after it.

    Then I fed the exact bytes from my first report through the parser on 53fed26, with tag_document registered, one character at a time:

    parser tool_calls content returned
    main at 53fed26 0 0 bytes
    the same, with #17284 0 137 bytes, the whole <tool_call> block

    So the fix still does what it says on current main, and the failure it addresses is still live on the current release.

    @aldi949 the case is already reduced to a single request. The exact JSON is in the issue body and it reproduces on a 0.5b model. You can run Doctor against it directly, without my setup. The result it should reach is above: the model puts the tool description in name, no registered tool matches, and the buffered output is dropped.

  12. rick-github commented on Sep 14, 2026

    @rick-github
    Collaborator

    A simple fix is just to have the model use the embedded Jinja template, which will then result in the runner using constrained generation.

    ollama create qwen2.5:7b-instruct-jinja -f <(ollama show --modelfile qwen2.5:7b-instruct | egrep "^(FROM|SYSTEM|PARAMETER)")
    $ curl -s localhost:11434/api/chat -d '{
      "model": "qwen2.5:7b-instruct-jinja",
      "messages": [
        {
          "role": "user",
          "content": "Tag document doc-17 with exactly alpha, beta, and gamma in that order."
        }
      ],
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "tag_document",
            "description": "Apply tags to a document.",
            "parameters": {
              "type": "object",
              "required": ["document_id", "tags"],
              "properties": {
                "document_id": { "type": "string" },
                "tags": { "type": "array", "items": { "type": "string" } }
              }
            }
          }
        }
      ],
      "tool_choice": "auto",
      "stream": false,
      "temperature": 0.0,
      "top_p": 1.0,
      "seed": 42,
      "max_tokens": 1024
    }' | jq
    {
      "model": "qwen2.5:7b-instruct-jinja",
      "created_at": "2026-09-14T14:49:29.150232134Z",
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "GjCGCBLf8x55VriRkvQhjJDzXxd75B5X",
            "function": {
              "index": 0,
              "name": "tag_document",
              "arguments": {
                "document_id": "doc-17",
                "tags": [
                  "alpha",
                  "beta",
                  "gamma"
                ]
              }
            }
          }
        ]
      },
      "done": true,
      "done_reason": "stop",
      "total_duration": 3165620128,
      "load_duration": 1311705314,
      "prompt_eval_count": 204,
      "prompt_eval_cached_count": 203,
      "prompt_eval_duration": 40675000,
      "eval_count": 37,
      "eval_duration": 1460246000
    }

    If using modified models is not acceptable, OLLAMA_GO_TEMPLATE=0 can be set in the server environment and all models will use their embedded Jinja template if a PARSER is not configured.

    $ OLLAMA_GO_TEMPLATE=0 ollama serve
    $ curl -s localhost:11434/api/chat -d '{
      "model": "qwen2.5:7b-instruct",     
      "messages": [
        {
          "role": "user",
          "content": "Tag document doc-17 with exactly alpha, beta, and gamma in that order."
        }
      ],
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "tag_document",
            "description": "Apply tags to a document.",
            "parameters": {
              "type": "object",
              "required": ["document_id", "tags"],
              "properties": {
                "document_id": { "type": "string" },
                "tags": { "type": "array", "items": { "type": "string" } }
              }
            }
          }
        }
      ],
      "tool_choice": "auto",
      "stream": false,
      "temperature": 0.0,
      "top_p": 1.0,
      "seed": 42,
      "max_tokens": 1024
    }' | jq
    {
      "model": "qwen2.5:7b-instruct",
      "created_at": "2026-09-14T15:00:24.3576331Z",
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "mvkyjHfVgF31ZH7zakol2PILhza0osRD",
            "function": {
              "index": 0,
              "name": "tag_document",
              "arguments": {
                "document_id": "doc-17",
                "tags": [
                  "alpha",
                  "beta",
                  "gamma"
                ]
              }
            }
          }
        ]
      },
      "done": true,
      "done_reason": "stop",
      "total_duration": 32574775358,
      "load_duration": 30343703421,
      "prompt_eval_count": 204,
      "prompt_eval_cached_count": 0,
      "prompt_eval_duration": 835543000,
      "eval_count": 37,
      "eval_duration": 1385923000
    }
  13. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    Confirmed here on qwen2.5:0.5b. Both of your routes produce the tool call.

    server model tool_calls content bytes eval_count
    default qwen2.5:0.5b 0 0 40
    default qwen2.5:0.5b-jinja 1 0 37
    OLLAMA_GO_TEMPLATE=0 qwen2.5:0.5b 1 0 37

    I built the jinja variant with your exact recipe.

    That is a working route today, and it is the answer I would give someone hitting this in production.

    It does not cover what the issue reports. Row 1 generated 40 tokens and returned 0 bytes. The caller sees content: "", done_reason: stop, and no error. Nothing tells it that output was produced and then dropped.

    Constrained generation stops this model from putting an unmatched name in that field. It does not change what the Go template path does when a name still does not match. That is the part #17284 changes: return the buffered text as content instead of discarding it.

    Any model on the Go template path can still emit a name that matches no registered tool. When it does, the bytes go nowhere.

  14. rick-github commented on Sep 14, 2026

    @rick-github
    Collaborator

    So don't use the Go template. I'm not privy to the roadmap that the maintainers have, but the CGO engine has been removed. Moving forward, ollama is likely to transition into a model manager/scheduler for the various runners - llama-server, MLX. No models added to the library since the switch to 0.30.0 include a Go template, so it seems that legacy support will continue but the road ahead is either PARSER or Jinja.

  15. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    Agreed on direction. Nothing in #17284 adds to the Go path or slows the move to PARSER and Jinja.

    Row 1 of my table is a plain ollama pull qwen2.5:0.5b. No template I wrote, no flag. That model is in the library today and it takes the Go path. The caller did not choose that template.

    What the path does now is the part I want to fix. It buffers the output, finds no tool with that name, and drops the bytes. The caller sees content: "", done_reason: stop, and no error. Nothing in the response says text was produced.

    #17284 changes 5 lines in tools/tools.go. It returns the buffered text as content instead of discarding it. Legacy support that silently loses output is worse than legacy support that returns it.

  16. rick-github commented on Sep 14, 2026

    @rick-github
    Collaborator

    Or, set OLLAMA_GO_TEMPLATE=0 and get an actual tool call, not text that the client needs to interpret.

  17. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    OLLAMA_GO_TEMPLATE=0 does give a tool call. I measured it again on 0.34.0 just now, and it is row 3 of the table above.

    You are right that #17284 returns raw text. The content it returns is the <tool_call> block, and the client has to read it.

    The part I am arguing is what happens when nobody sets that variable. envconfig.Var reads it from the server process environment. There is no per-request or per-model override in the API. A caller who does not run the server cannot set it.

    The server already makes this choice on its own. shouldPreferChatTemplate in server/images.go switches to the GGUF chat template when the Go template cannot round-trip tools. For qwen2.5:0.5b both templates report [completion tools], and the Go template passes goTemplateHasToolRoundTrip, so the server keeps the Go one.

    Two runs on 0.34.0, same request, same seed, from the server's own template selection log line:

    server env selected tool_calls content bytes eval_count
    default go_template 0 0 40
    OLLAMA_GO_TEMPLATE=0 gguf_chat_template 1 0 37

    Row 1 is what this model does today with no environment change. It generates 40 tokens, returns zero bytes, and reports done_reason: stop with no error.

  18. aldi949 commented on Sep 14, 2026

    @aldi949

    @devYRPauli Thanks — I ran the exact issue-body request against Ollama 0.34.0 + qwen2.5:0.5b.
    I reproduced the silent-drop behavior on /api/chat 2/3 times, but not stably 3/3. On /v1/chat/completions I got 0/3 reproductions of the target failure class in my environment.
    Before I adapt ToolCall Doctor to support this failure predicate, could you share the exact Ollama launch command, model state/config, and request command you used for your stable reproduction?
    I want to match your environment exactly rather than accidentally validate a nearby failure variant.

  19. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    Here is the exact setup.

    Server: ollama serve with no environment variables set. Version 0.34.0, from curl -s http://127.0.0.1:11434/api/version.

    Model: qwen2.5:0.5b, ID a8b0c5157701, blob sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515, Q4_K_M, 494.03M parameters. The issue body names qwen2.5:7b-instruct. My table above moved to 0.5b, so use 0.5b to match my numbers.

    Your 2/3 on /api/chat has a cause. /api/chat does not read temperature, top_p or seed at the top level. ChatRequest in api/types.go declares no such fields. Sampling parameters reach the runner only through Options. A top-level "seed": 42 is dropped by the JSON decode with no error, so the run is unseeded.

    Five runs of each shape, same server, same model, nothing else changed:

    where the sampling params sit tool_calls per run eval_count per run
    top level 1, 0, 0, 1, 0 37, 40, 37, 37, 37
    inside options 0, 0, 0, 0, 0 40, 40, 40, 40, 40

    The top-level shape is unseeded and drops the call 3 times in 5. The options shape is seeded and drops it 5 times in 5. Use the second one as the predicate.

    curl -s http://127.0.0.1:11434/api/chat -d '{
      "model": "qwen2.5:0.5b",
      "messages": [{"role": "user", "content": "Tag document doc-17 with exactly alpha, beta, and gamma in that order."}],
      "tools": [{"type": "function", "function": {"name": "tag_document", "description": "Apply tags to a document.", "parameters": {"type": "object", "required": ["document_id", "tags"], "properties": {"document_id": {"type": "string"}, "tags": {"type": "array", "items": {"type": "string"}}}}}}],
      "stream": false,
      "options": {"temperature": 0.0, "top_p": 1.0, "seed": 42}
    }'

    The failure is message.tool_calls absent, message.content empty, eval_count 40, done_reason stop.

    /v1/chat/completions reproduces 5 times in 5 here, so I cannot match your 0/3. openai/openai.go copies top-level temperature, seed and top_p into options, so the request shape is not the difference. I ran the issue-body JSON with the model changed to qwen2.5:0.5b and got tool_calls 0, content 0 bytes, completion_tokens 40, finish_reason stop, on all five.

    One line in the server log tells you whether you are on the same path:

    msg="template selection" model=registry.ollama.ai/library/qwen2.5:0.5b selected=go_template renderer="" parser="" go_template="[completion tools]" chat_template="[tools completion]" harmony=null renderer_parser=null
    

    If yours says selected=gguf_chat_template, the server picked the other template and you get a parsed tool call. That is row 2 of my table above, and it is not the failure you want to target.

  20. aldi949 commented on Sep 14, 2026

    @aldi949

    @devYRPauli Thanks — I matched the setup more closely and got a different result.

    Verified:

    • Ollama 0.34.0
    • qwen2.5:0.5b
    • plain ollama serve
    • no relevant env overrides
    • selected=go_template
    • sampling params inside options

    But both routes now produce a valid tag_document tool call 5/5:

    /api/chat

    • tool_calls: present 5/5
    • eval_count: 37
    • done_reason: tool_calls

    /v1/chat/completions

    • tool_calls: present 5/5
    • completion_tokens: 37
    • finish_reason: tool_calls

    So I still cannot reproduce your 40-token silent-drop state.

    Could you share the exact output of:

    ollama show qwen2.5:0.5b --modelfile

    and, if possible, the exact Ollama build/commit info plus OS/architecture?

    I want to diff the model/template metadata and runner environment against mine, because the request path is now confirmed identical.

  21. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    Reproduced again on the same box, 5/5, with your setup: 0.34.0, qwen2.5:0.5b, plain ollama serve, no env overrides, selected=go_template, sampling inside options.

    Here is the measurement that separates our runs. The tokens are generated and then dropped.

    request prompt_eval_count eval_count tool_calls content
    /api/chat with tools 177 40 none ""
    /api/chat, same schema pasted into a system message, no tools key 177 37 n/a the full block below

    Same endpoint, same model, same options, same prompt length. Without the tools key the server returns this as content:

    <tool_call>
    {"name": "tag_document", "arguments": {"document_id": "doc-17", "tags": ["alpha", "beta", "gamma"]}}
    </tool_call>
    

    With the tools key it returns content: "", tool_calls: null, done_reason: "stop", eval_count: 40. Streamed, that request produces exactly one chunk, the final done chunk, with empty content. So the runner generates 40 tokens and none of them reach the caller.

    The drop does not depend on the schema. Four variants, all with tools present:

    schema change prompt_eval_count eval_count result
    baseline 177 40 empty
    long description inside items 197 49 empty
    long description on document_id 197 40 empty
    items removed 171 40 empty

    eval_count moves with the prompt, so the model generates real output every time.

    Your 37 is my no-parser number, which points at the backend. Mine:

    msg="inference compute" library=Metal name=MTL0 description="Apple M1 Pro" type=iGPU total="11.8 GiB"
    system_info: MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | ACCELERATE = 1 | REPACK = 1
    load_tensors: offloading 23 repeating layers to GPU
    

    macOS 26.6.2 (build 25G83), Darwin 25.6.0, arm64, Apple M1 Pro, 16 GiB, ollama 0.34.0 installed from Homebrew. What does the inference compute line say on yours?

    A 0.5B Q4_K_M model at greedy decoding can flip an argmax between backends, and that changes the continuation. That accounts for 37 against 40 on the same prompt.

    The backend only decides who lands on it. The defect is what happens next. When the output does not match the parser, ollama returns an empty assistant message with done_reason: "stop" and no error. A caller cannot tell that apart from a model that chose to say nothing, and the 40 tokens are gone.

    ollama show qwen2.5:0.5b --modelfile gives the stock qwen2.5 chat template, SYSTEM You are Qwen, created by Alibaba Cloud. You are a helpful assistant., and no PARAMETER lines at all. The blob is sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515, Q4_K_M, 494.03M. Worth diffing that blob digest against yours.

  22. aldi949 commented on Sep 14, 2026

    @aldi949

    @devYRPauli Thanks — that isolates the remaining difference very clearly. My local reproduction is CPU, while yours is Metal, and everything else now matches closely.
    Since you already have the target failure stable 5/5, would you be willing to run ToolCall Doctor against the reproducer on your machine? I can give you the exact command/setup so it should only take a few minutes.
    That would let us test Doctor against the real failure path instead of trying to reproduce Metal behavior on a different backend.

  23. devYRPauli commented on Sep 14, 2026

    @devYRPauli
    Author

    Doctor can be tested against this without my machine. The drop is in the parser, and it reproduces with no model, no server and no backend.

    tools/tools.go:394 on main at 53fed26:

    func (p *Parser) Content() string {
    	if p.n > 0 {
    		return ""
    	}
    
    	if p.tag == "{" || p.tag == "[" {
    		return string(p.buffer)
    	}
    
    	return ""
    }

    p.n counts parsed tool calls. When zero calls parse, a { or [ tag hands the buffer back and a string tag returns an empty string.

    I fed the recovered block from the issue body to NewParserWithTag once per tag, with the tool registered as tag_document:

    ref tag calls Content()
    main 53fed26 <tool_call> 0 0 bytes
    main 53fed26 { 0 125 bytes
    #17284 head 72413e5 <tool_call> 0 137 bytes
    #17284 head 72413e5 { 0 125 bytes

    The buffer holds 137 bytes in every row. Only the string-tag row on main throws them away.

    #17284 deletes that special case, and the whole tools package passes on its head. GitHub reports it mergeable and 1 commit behind main.

    Metal only decides whether the model emits a name that matches no registered tool. After that the loss is deterministic and the backend does not matter. So the predicate Doctor can target is: the tag was found, zero calls parsed, and the tag is not { or [.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions