Skip to content

[Fix] Honor ChatTemplate instances in load_template_class - #1524

Open
BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-chat-template-instance-discarded
Open

BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-chat-template-instance-discarded

Conversation

@BlueX888

Copy link
Copy Markdown

Summary

load_template_class now uses the class of a ChatTemplate instance passed as chat_template, instead of discarding it and falling back to ChatML.

Why

load_template_class (guidance/chat.py:47) only handled chat template classes (guidance/chat.py:57) and template strings (guidance/chat.py:62). A ChatTemplate instance matched neither branch, so it fell through to the ChatML default (guidance/chat.py:80) with a "was unable to be loaded" warning, replacing the caller's template with ChatML.

The public API annotates the parameter as str | ChatTemplate | None (guidance/models/_engine/_tokenizer.py:36, also guidance/models/_transformers.py:61 and guidance/models/_llama_cpp.py:62), and the onnxruntime tutorial passes an instance (chat_template=Phi4MiniChatTemplate() in notebooks/tutorials/onnxruntime_models.ipynb). With that documented usage, Phi-4-mini prompts were formatted with ChatML markers (<|im_start|>user) instead of the model's own markers (ammad).

Changes

  • guidance/chat.py: in load_template_class, if chat_template is a ChatTemplate instance, use its class so it is handled by the existing class branch; documented the added precedence in the docstring.
  • tests/unit/test_chat.py: new regression tests covering load_template_class with an instance (must not warn, must preserve role markers) and a Tokenizer constructed with an instance.

Testing

New tests on the unpatched tree (both fail — the instance falls through to the ChatML fallback):

$ python -m pytest tests/unit/test_chat.py -v
FAILED tests/unit/test_chat.py::test_load_template_class_from_instance - UserWarning: Chat template <guidance.chat.Phi4MiniChatTemplate object ...> was unable to be loaded directly into guidance. Defaulting to the ChatML format ...
FAILED tests/unit/test_chat.py::test_tokenizer_chat_template_from_instance - AssertionError: assert '<|im_start|>user\n' == 'ammad'
2 failed, 1 warning in 0.01s

New tests on the patched tree:

$ python -m pytest tests/unit/test_chat.py -v
tests/unit/test_chat.py::test_load_template_class_from_instance PASSED   [ 50%]
tests/unit/test_chat.py::test_tokenizer_chat_template_from_instance PASSED [100%]
2 passed in 0.01s

Existing unit suite (no regressions):

$ python -m pytest tests/unit/ -q
1639 passed, 1 skipped, 220 xfailed, 1 warning in 11.45s

Format and lint (per CONTRIBUTING.md):

$ ruff format --check guidance/chat.py tests/unit/test_chat.py
2 files already formatted
$ ruff check guidance/chat.py tests/unit/test_chat.py
All checks passed!

A ChatTemplate instance passed as chat_template (as the onnxruntime
tutorial and the str | ChatTemplate | None annotations suggest) fell
through to the ChatML fallback, replacing the user's template with
ChatML despite the warning. Use the instance's class so it is treated
like a passed-in class.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant