Skip to content

[Fix] Handle tokenizers without a BOS token in Engine.chat_completion - #1545

Open
BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-chat-completion-bos-token-none-attributeerror
Open

BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-chat-completion-bos-token-none-attributeerror

Conversation

@BlueX888

Copy link
Copy Markdown

Summary

Engine.chat_completion raised AttributeError for tokenizers that have no BOS token (e.g. Qwen2/Qwen3, where bos_token_id is None). It now renders the chat template with an empty bos_token in that case.

Why

chat_completion_streaming built the chat-template context by unconditionally decoding the BOS token:

tokens = {
    "eos_token": self.tokenizer.eos_token.decode("utf-8"),
    "bos_token": self.tokenizer.bos_token.decode("utf-8"),
}

Tokenizer.bos_token is bytes | None and returns None when bos_token_id is None (guidance/models/_engine/_tokenizer.py:59-63), so the .decode() call raised AttributeError: 'NoneType' object has no attribute 'decode' before the chat template was ever rendered. mypy reports the same problem on that line (Item "None" of "bytes | None" has no attribute "decode").

This is inconsistent with the rest of the engine: Parser.process_prompt already guards on self.tokenizer.bos_token_id is not None (guidance/_parser.py:81-84), and the default ChatML template does not reference bos_token, so no BOS is needed to render a prompt.

The existing test_chat_completion runs against gpt2, whose tokenizer has bos_token_id=50256, so the None path was untested.

Changes

  • guidance/models/_engine/_engine.py: pass "" as bos_token to the chat template when Tokenizer.bos_token is None.
  • tests/model_integration/test_engine.py: add test_chat_completion_tokenizer_without_bos_token, which loads the Qwen3-0.6B tokenizer (no BOS token) and runs chat_completion end to end against a stub engine. Only the tokenizer is downloaded, mirroring the tokenizer-only download pattern already used in tests/model_integration/test_tokenizers.py.

Testing

# regression test, unpatched tree
HF_HUB_OFFLINE=1 python -m pytest tests/model_integration/test_engine.py::test_chat_completion_tokenizer_without_bos_token -x -q
# 1 failed in 2.03s — AttributeError: 'NoneType' object has no attribute 'decode' (guidance/models/_engine/_engine.py:671)

# regression test, patched tree
HF_HUB_OFFLINE=1 python -m pytest tests/model_integration/test_engine.py::test_chat_completion_tokenizer_without_bos_token -q
# 1 passed in 1.98s

# existing tests for the same module (includes test_chat_completion on gpt2)
python -m pytest tests/model_integration/test_engine.py -q
# 2 passed in 3.20s

# broader suite
python -m pytest tests/unit -q
# 1637 passed, 1 skipped, 220 xfailed, 1 warning in 16.13s

# format / imports / lint (as run by code_quality.yml)
ruff format --check               # 150 files already formatted
ruff check --select I,RUF022      # All checks passed!
ruff check                        # All checks passed!

# mypy guidance (informational job)
mypy guidance
# Found 125 errors in 24 files (checked 72 source files) — unchanged pre-existing set;
# the unpatched bos_token line was itself the 126th error, resolved by this fix

chat_completion_streaming unconditionally decoded tokenizer.bos_token,
which is None when bos_token_id is None (e.g. Qwen2/Qwen3 tokenizers),
so chat_completion raised AttributeError before rendering the chat
template. Pass an empty string to the template in that case instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant