Gemma 4 tool calling support (parser availability & required configuration)

Hello,

I am currently integrating Gemma 4 (via vLLM) with Hermes Agent and encountering issues with tool calling parsing.

Context

Setup:

Model: google/gemma-4-E4B-it
Runtime: vLLM (vllm-openai:gemma4)
Flags:
- --tool-call-parser gemma4
- --enable-auto-tool-choice
Hermes Agent (latest version)

Observed behavior

Gemma 4 produces tool calls in the following format:

<|tool_call>call:search_files{pattern:<|"|>/var/log<|"|>,target:<|"|>files<|"|>}<tool_call|>

However:

The tool_calls field is empty
The tool call remains embedded in message.content
Hermes does not execute the tool, as it relies on message.tool_calls

This results in a silent failure of tool execution.

This behavior appears consistent with other reports where:

Gemma outputs valid tool calls
but parsers fail to extract them into structured fields ([GitHub]1)

Questions

Does Hermes Agent currently support a native Gemma parser?
- I could not find any gemma parser in environments/tool_call_parsers
- Is this planned or already available under another name?
Is there a recommended parser for Gemma 4?
- pythonic seems partially compatible with call:func{...} formats
- but Gemma 4 uses <|tool_call> ... <tool_call|> tokens
Is a custom chat_template required for Gemma?
- vLLM documentation suggests that only some models have built-in tool templates
- Gemma appears to require explicit formatting ([vLLM]2)
Is Hermes expected to handle raw tool call text as fallback?
- Currently Hermes only checks message.tool_calls
- Would a fallback parsing mechanism (from content) be considered?

Goal

Clarify the recommended way to use Gemma 4 + vLLM with Hermes, and whether first-class support is planned.

Thanks!

feat: add Gemma 4 tool call parser

If you're running Gemma 4 locally on Mac, Rapid-MLX solves this server-side — it has a native Gemma 4 tool call parser that converts Gemma's <|tool_call>call:name(args)<tool_call|> format into standard OpenAI tool_calls JSON before it reaches Hermes.

Hermes doesn't need any client-side parser — it just sees normal structured tool calls.

We tested the full pipeline: 20/20 integration tests pass on Gemma 4 26B MoE, including multi-step tool calling, file editing (patch), code generation + execution, and stress testing (0 tag leaks).

Setup:
```bash
pip install vllm-mlx
rapid-mlx serve gemma-4-26b # auto-downloads + auto-configures parser
```

Then in `~/.hermes/config.yaml`:
```yaml
model:
provider: "custom"
base_url: "http://localhost:8000/v1"
```

Test suite: test_hermes.py (20 tests covering all agentic workflows)

Duplicate of #7457 — Gemma 4 tool calling support. A client-side parser is already submitted in PR #7449.

Duplicate of #7457

This issue appears to describe the same problem as #7457.
Maintainer might want to consolidate or pick one as canonical.

I would argue that this is not a duplicate of Issue #7457 on the basis that:

that issue talks specifically about the Nous Inference API
this issue states the broader problem—regardless of the API endpoint, Gemma 4 tool calling isn’t reliably handled by Hermes Agent at this time.

Reportedly related pull requests:

fix(agent): recover gemma4 tool_call.arguments corrupted by leaked chat-template markers #19887 remains open but in my opinion is unrelated as it talks specifically about vLLM.
feat: add Gemma 4 tool call parser #7449 was closed but hasn't solved the problem for me:

So from what I can tell, at this stage there is no proposed fix.

Quick state check on the related threads:

#7457 is OPEN. That issue is the narrower Nous Inference API/server-side parsing track.
#7449 is CLOSED and was explicitly superseded; its old parser target (environments/tool_call_parsers/) no longer exists on main, so that PR is not the path forward anymore.
#19887 is OPEN and is the current Hermes-side fix in flight. It moves the recovery logic to the modern path in agent/transports/chat_completions.py::normalize_response() and is intended to handle Gemma 4 tool-call corruption on current main.

So there is an active proposed fix, but it is #19887, not #7449. If your failing case is the same family of Gemma 4 parsing/normalization failures on current Hermes, #19887 is the thread to watch and add concrete repro details to.

TL;DR: Same Gemma 4 tool-calling failure as this issue, but on
oMLX (Apple-Silicon local server) rather than vLLM. I ran a controlled
bisection to find out when Gemma 4 breaks its tool syntax. Three results
worth having in this thread: (1) the exact context conditions that trigger
the degradation, with verbatim raw output; (2) why the in-flight Hermes fix
(#19887) cannot help when the server strips the malformed call — the two
fixes needed are complementary, oMLX side filed at jundot/omlx#1846; (3) a
configuration workaround that took a 26B kanban worker from 16 consecutive
protocol failures to a clean run, today, with no code change.

Setup

Hermes Agent kanban worker → oMLX v0.4.4rc1 →
gemma-4-26B-A4B-it-MLX-4bit (macOS, 64 GB). Standard Hermes worker
loadout: 36 tool schemas (~80 KB JSON) + system prompt (~34 KB) with
KANBAN_GUIDANCE and the skills index.

The worker always completes its first kanban_show call fine, then dies on
the next call: Hermes receives empty content and empty tool_calls,
nudges, gets the same emptiness, and the worker exits without
kanban_complete/kanban_block (protocol_violation) — 16 runs in a row.

What the model actually emits at the failure point

Captured raw (no chat parser) by re-rendering the exact failing
conversation through the model's own chat_template.jinja and replaying it
on /v1/completions:

<think>
</think>
<|tool_call>call:todo(todos=[{content:<|"|>Clarify goal and draft labor graph for t_77d3100f<|"|>,id:<|"|>todo-1<|"|>,status:<|"|>pending<|"|>}])<tool_call|>

Canonical envelope, canonical <|"|> quoting — but parenthesized
Python-style kwargs instead of the curly call:name{...} form the chat
template defines. oMLX's gemma4 parser doesn't recognize the variant and
silently strips the whole block, which is why Hermes sees nothing at all.

When it breaks (bisection, direct API, n≥6 per cell)

Ruled out: prompt length alone (22.5k tokens of neutral padding + 1 tool →
clean), tool count alone (36 schemas + short prompt → clean), nested-args
capability (the todo schema alone → 3/3 clean). What's left is the
combination every agent framework creates — an instruction-dense system
prompt plus a large schema surface:

Same turn-2 worker payload, one factor varied	Well-formed rate
36 tool schemas	0/8
7 schemas (kanban toolset only)	5/6
36 schemas, system prompt minus KANBAN_GUIDANCE	7/8

Why #19887 alone won't cover this backend

#19887 repairs corrupted arguments that reach Hermes. On oMLX nothing
reaches Hermes — the server deletes the unparsable block server-side, so
content and tool_calls are both empty and there is nothing to recover. I
filed jundot/omlx#1846 asking oMLX to either fallback-parse the
parenthesized variant or pass unparsable markup through in
message.content. If oMLX passes it through, a Hermes content-side
fallback would then have something to work with — the two fixes complement
each other.

Workaround that works today (and a feature suggestion)

Shrinking the schema surface flips the model back to canonical syntax. In
the worker profile's config.yaml:

platform_toolsets:
  cli: [kanban]

(Note: the top-level toolsets: key is additive to the platform default,
so it does not reduce the surface — that distinction cost me a failed
experiment.)

With that one change, the same 26B worker ran 14 consecutive correctly
parsed tool turns and exited with a clean, reasoned kanban_block —
first protocol-complete run after 16 failures. A documented per-profile
"lite loadout / context budget" for local models (fewer schemas, leaner
injected blocks) would make small local models viable kanban workers
without any parser change.

Thanks, this is useful because it separates two different failure modes.

#7457 is CLOSED (completed). That was the narrower Nous Inference API/server-side parsing track.
#7449 is CLOSED (not merged / superseded). Its old parser path is no longer the route on current main.
#19887 is OPEN. That PR is still the current Hermes-side recovery path, but only for the complementary case where malformed Gemma output still reaches Hermes and can be normalized there.

Your oMLX repro is different: if the server strips the malformed tool block entirely and Hermes receives empty content plus empty tool_calls, Hermes has nothing to recover. So this does not contradict #19887; it narrows its scope. The upstream oMLX parser issue you filed is the load-bearing fix for that backend-side variant.

The Hermes-specific part of your report is the workaround / product gap: reducing the injected schema surface (platform_toolsets.cli: [kanban]) makes small local Gemma workers materially more reliable. If maintainers want to act on something in Hermes from this thread, that looks more like a focused "lite loadout / smaller context budget for local workers" feature than another parser-only fix.

Can someone help confirm this to NOT be a quantization issue (ideally 4-bit, sometimes larger)?

This was generated by AI during triage.

Re-triage note: this is not being closed as a duplicate of #7457 (now CLOSED, and scoped to the Nous Inference API), and the previously-cited parser PR #7449 is closed/superseded (its environments/tool_call_parsers/ path no longer exists on main). Keeping this open as the broader any-endpoint Gemma 4 tool-call parsing thread. Current in-flight Hermes-side work: #19887 (gemma4 tool_call.arguments recovery) and #47074 (Gemma/oMLX text tool-call recovery). Related: #7457, #19887, #47074.

Short answer: not confirmed as a quantization-only problem from the evidence in this thread. The strongest signal so far is that the failure correlates with backend/parser behavior and prompt/schema pressure, not just 4-bit by itself. In the oMLX repro above, the same gemma-4-26B-A4B-it-MLX-4bit model stopped failing when the injected tool/schema surface was reduced, which points to a context-formatting / parser interaction rather than a clean "4-bit is broken" conclusion.

Current related thread states:

#7457 is CLOSED and was the narrower Nous Inference API track.
#7449 is CLOSED / superseded and is not the path on current main.
#19887 is OPEN and covers the Hermes-side recovery case where malformed Gemma tool-call data still reaches Hermes.
#47074 is OPEN and is the closer match for the text-side Gemma/oMLX recovery path.

So today I would phrase it as: this is not yet ruled in or ruled out as a quantization contributor, but the thread does already show a non-quantization variable that materially changes the outcome. If someone wants to pin quantization down, the useful data would be the same prompt/tool payload run across the same backend with 4-bit vs larger quantizations, while keeping the schema surface fixed.

This was generated by AI during triage.

Summary

One PR, #7449, directly addresses #6626 by adding a Gemma 4 raw-text parser and marker-based fallback, including the reported brace syntax and special quote tokens. Its integration point is limited to the former RL environment path, so it does not fix the production CLI path or structured-arguments corruption described in the discussion.

Related pull requests

feat: add Gemma 4 tool call parser #7449 [closed] best fix — (+406/-6) — partial fix, closed/superseded: adds and tests a Gemma 4 parser for raw tool-call text, including the exact Gemma 4 tool calling support (parser availability & required configuration) #6626 syntax, but wires fallback parsing through environments/agent_loop.py rather than the production normalize_response() path; the target parser directory was subsequently removed from main, and the diff does not handle markers leaking into already-structured tool_call.arguments.

Suggested consolidation

Do not merge or reopen #7449 in its current form despite its selection as the verify-stage best fix: the diff validates useful parsing behavior, but its registration and invocation paths no longer exist on current main and contributor discussion identifies the modern transport normalization path as the required integration point. Keep #7449 closed as a superseded reference implementation and consolidate any retained parser cases into current-main work such as #19887/#47074, with coverage for both raw content recovery and corrupted structured arguments.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I6626(["issue #6626 (open)"])
    P7449["PR #7449 (closed)"]
    MoreI(["+1 more issues not shown"])
    P7449 -->|best fix| I6626
    class I6626 open
    class I6626 target
    class P7449 closed
    class P7449 best
    class MoreI closed
    click I6626 "https://github.com/NousResearch/hermes-agent/issues/6626"
    click P7449 "https://github.com/NousResearch/hermes-agent/pull/7449"

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

Cross-PR triage: Reviewed 1 pull request and 2 issues in this complex. Each diff was read against this issue; Assessment working set: 19 kB of PR diffs, 8 kB of issue/PR text, 15 kB of discussion (19 comments), 3 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

Additional reproduction: Gemma 4 + LM Studio leaks tool-call control tokens into `write_file` path arguments

I am encountering a related tool-calling failure with Hermes Agent running locally on Omarchy Linux.

Environment

Agent: Hermes Agent, terminal.backend=local
Inference server: LM Studio (http://127.0.0.1:1234/v1)
Model: gemma-4-12b-obliterated
Model size: 7.38 GB
Context: 64,000 tokens
Tool: Hermes built-in write_file
Destination: An existing, writable Obsidian vault

Observed behavior

Hermes attempted to create a Markdown document, but the generated write_file arguments contained unexpected tool-call protocol markers appended directly to the destination filename.

Attempt 1 — corrupted filename suffix:

Omarchy Tailscale Configuration.md followed by:

`}<tool_call|></ul><|channel>thought

<channel|><|tool_call>call:terminal{command:

Attempt 2 — corrupted filename suffix:

Omarchy Tailscale Configuration.md followed by:

`}<tool_call|><|channel>thought

<channel|><|tool_call>call:read_file{path:

Both attempts created actual files with corrupted filenames.

Hermes subsequently returned:

Post-write verification failed ... on-disk content hash differs from the intended write.

Database evidence

I inspected Hermes' SQLite conversation database:

~/.hermes/state.db

Specifically, I examined the messages.tool_calls records for the affected session.

Both recorded assistant tool calls contained the malformed strings inside the serialized write_file arguments, before the filesystem tool executed.

The tool-result records immediately afterward contained the verification errors.

Importantly, the arguments were syntactically valid JSON. The corruption was inside the path string itself, so ordinary JSON parsing did not detect the problem.

Independent LM Studio API tests

I tested the same model and destination path directly through LM Studio's /v1/chat/completions endpoint, bypassing Hermes.

Both tests used a simulated write_file(path, content) function schema and tool_choice="required".

Each isolated API mode was tested once. These results demonstrate that simple tool calls work, but do not establish reliability under Hermes' full prompt, conversation history, and tool schemas.

Relevant Hermes implementation

The following code paths appear relevant:

agent/chat_completion_helpers.py — streaming argument accumulation
agent/conversation_loop.py — JSON argument validation
tools/file_operations.py — file-writing and post-write verification

The existing JSON validation accepts a syntactically valid argument even when the path contains unintended model-control tokens.

Distinction from other reported failures

This is not simply a case where tool-call text appears in message.content instead of message.tool_calls.

In this reproduction, message.tool_calls was populated, and the structured function.arguments.path value itself contained the leaked control syntax.

This may be related to the argument-corruption behavior discussed in PR #19887.

Questions

Is this a known compatibility issue between LM Studio's Gemma 4 tool-call parser and Hermes?
Is there a recommended way to capture the raw LM Studio response and compare it with the arguments persisted by Hermes?
Is there a supported configuration or mitigation for local Gemma 4 models under Hermes' full tool-schema load?
Should Hermes reject recognizable protocol/control markers in filesystem path arguments before performing write, move, or delete operations?

I can provide sanitized database records or run targeted reproductions if additional evidence would be helpful.

fix(agent): re-prompt on Gemma 4 pipe-delimited tool-call text (NousR…

Navigation Menu

Repository navigation

Gemma 4 tool calling support (parser availability & required configuration) #6626

Context

Observed behavior

Questions

Goal

Setup

What the model actually emits at the failure point

When it breaks (bisection, direct API, n≥6 per cell)

Why #19887 alone won't cover this backend

Workaround that works today (and a feature suggestion)

Summary

Related pull requests

Suggested consolidation

Complex graph

Additional reproduction: Gemma 4 + LM Studio leaks tool-call control tokens into `write_file` path arguments

Environment

Observed behavior

Database evidence

Independent LM Studio API tests

Relevant Hermes implementation

Distinction from other reported failures

Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gemma 4 tool calling support (parser availability & required configuration) #6626

Description

Context

Observed behavior

Questions

Goal

Activity

raullenchai commented on Apr 12, 2026

alt-glitch commented on Apr 29, 2026

alt-glitch commented on Apr 29, 2026

konsisumer commented on May 23, 2026

Lx commented on Jun 11, 2026

konsisumer commented on Jun 11, 2026

hugo-valcourt commented on Jun 12, 2026

Setup

What the model actually emits at the failure point

When it breaks (bisection, direct API, n≥6 per cell)

Why #19887 alone won't cover this backend

Workaround that works today (and a feature suggestion)

konsisumer commented on Jun 12, 2026

TomLucidor commented on Jul 8, 2026

alt-glitch commented on Jul 8, 2026

konsisumer commented on Jul 8, 2026

GottZ commented on Jul 29, 2026

Summary

Related pull requests

Suggested consolidation

Complex graph

80Builder80 commented on Oct 12, 2026

Additional reproduction: Gemma 4 + LM Studio leaks tool-call control tokens into write_file path arguments

Environment

Observed behavior

Database evidence

Independent LM Studio API tests

Relevant Hermes implementation

Distinction from other reported failures

Questions

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Additional reproduction: Gemma 4 + LM Studio leaks tool-call control tokens into `write_file` path arguments