fix: enable Gemma4 tool calling via mlx-vlm parser infrastructure#561
Closed
TipKnuckle wants to merge 1 commit intojundot:mainfrom
Closed
fix: enable Gemma4 tool calling via mlx-vlm parser infrastructure#561TipKnuckle wants to merge 1 commit intojundot:mainfrom
TipKnuckle wants to merge 1 commit intojundot:mainfrom
Conversation
TipKnuckle
force-pushed
the
feat/gemma4-tool-calling
branch
from
62bc4d0 to
a2480fc
Compare
Contributor
Author
|
Superseded by #565, which includes a complete rewrite of Gemma4 tool calling using a proper message extractor pattern. The approach here (delegating to mlx-vlm's tool parser infrastructure) turned out to be insufficient — the root cause was that the Gemma4 chat template has no handling for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes tool calling for Gemma4 VLM models by delegating to mlx-vlm's own parser infrastructure rather than mlx-lm's.
Problem
_inject_tool_callingwas importing_infer_tool_parserdirectly frommlx_lm.tokenizer_utils, which has no knowledge of Gemma4's<|tool_call>template marker. It returnedNonefor Gemma4, sohas_tool_callingwas never set and raw tool call XML was passed through into the context instead of being parsed.Change
Switch to mlx-vlm's own
_infer_tool_parserandload_tool_module(available since mlx-vlm commit43b9b20, already pinned by this project):mlx_vlm.tool_parsers._infer_tool_parsercalls mlx-lm's version first, then checks VLM-specific extras — including<|tool_call>→gemma4. Strictly a superset, no regressions for other models.mlx_vlm.tool_parsers.load_tool_moduleprefersmlx_vlm.tool_parsers.*overmlx_lm.tool_parsers.*, so Gemma4 gets the correct parser (right tokens, right escape format) rather than the unrelatedfunction_gemmaparser from mlx-lm.try/except ImportErrorguard means this degrades gracefully if run against an older mlx-vlm without thetool_parserspackage.Any future VLM model that mlx-vlm adds tool support for will be picked up automatically with no further changes needed here.