Skip to content

Commit 8d46973

Browse files
authored
Python: fix(python): keep ChatMessageContent.to_prompt a str when encoding is set (#14464)
### What this fixes `ChatMessageContent.to_prompt()` is annotated `-> str` but returns `bytes` for any message that carries an `encoding`. ```python root = self.to_element() return ElementTree.tostring(root, encoding=self.encoding or "unicode", short_empty_elements=False) ``` `encoding` is a real field on the content, so the `or` never protects it: setting `encoding="utf-8"` makes `tostring` take its bytes branch. Measured on `ca40aa72`: ``` encoding='utf-8' -> type=bytes value=b'<message role="user" encoding="utf-8">...' encoding='ascii' -> b'<?xml ...?>\n<message ...' # XML declaration too ``` This is user-visible rather than merely untidy: `jinja2_system_helpers._message_to_prompt` and its handlebars twin render a context variable via `str(context.to_prompt())`, so the prompt the model receives literally contains `b'<message role="user" ...>'` — bytes repr plus quote noise — whenever the message had an encoding set. Every other `tostring` call site in the same package already hardcodes `encoding="unicode"`: `chat_history.py:285`, `:317`, `:362`, `chat_message_content.py:275`, `handlebars_system_helpers.py:37`, `:55` and `jinja2_system_helpers.py:29`. This one line was the outlier, so the change aligns it rather than introducing a new convention. The message's own encoding is still preserved: it is an attribute on the element and round-trips through `to_element`/`from_element`; it describes the text content, not how we serialize a prompt string. ### Change One line, plus a docstring note, and a parametrized regression test over `utf-8`, `ascii` and `latin-1` asserting the prompt is a `str`, has no XML declaration, keeps the non-ASCII text, and re-parses to the same content. ### How this was verified `cd /private/tmp/sk/tree/python`, `PYTHONPATH=.`, the repo's own test file `tests/unit/contents/test_chat_message_content.py`. Base control — `semantic_kernel/contents/chat_message_content.py` restored from `ca40aa7226531d28a721d0ca0e451d0aaf86dafc` with `git diff -- <that file>` printing nothing (so the run measures upstream code with the new tests kept), sha re-echoed: ``` FAILED tests/unit/contents/test_chat_message_content.py::test_cmc_to_prompt_with_encoding[utf8] FAILED tests/unit/contents/test_chat_message_content.py::test_cmc_to_prompt_with_encoding[ascii] FAILED tests/unit/contents/test_chat_message_content.py::test_cmc_to_prompt_with_encoding[latin1] 3 failed, 43 passed ``` With the patch applied: `46 passed`. Enclosing directory, both revisions, same command (`pytest tests/unit/contents`): the `test_chat_history.py::test_to_from_file` / `::test_chat_history_serialize` collection errors appear **identically at base and at head** — they come from optional extras missing in my environment, not from this diff. I did not run the integration or dotnet suites. `ruff check` on the two files reports only a pre-existing `D` finding at `chat_message_content.py:201`, present at base too; `ruff format --check` is clean. (`mypy` was not installed in this environment, so the CI type gate is unverified here.) Dedup: searched issues and PRs (all states) for `to_prompt`, `message_to_prompt` and `encoding unicode prompt`; the only hits are merged history (#13738, #5550). The nearby `data_uri` MIME-subtype defect I found while investigating is already claimed by open #14304, so it is deliberately not part of this PR.
1 parent 07ffe35 commit 8d46973

2 files changed

Lines changed: 23 additions & 1 deletion

File tree

‎python/semantic_kernel/contents/chat_message_content.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -294,11 +294,15 @@ def from_element(cls, element: Element) -> "ChatMessageContent":
294294
def to_prompt(self) -> str:
295295
"""Convert the ChatMessageContent to a prompt.
296296
297+
The encoding attribute of the content is not used here,
298+
this always returns a string, the encoding is only relevant
299+
for the actual text content, not for the prompt representation.
300+
297301
Returns:
298302
str - The prompt from the ChatMessageContent.
299303
"""
300304
root = self.to_element()
301-
return ElementTree.tostring(root, encoding=self.encoding or "unicode", short_empty_elements=False)
305+
return ElementTree.tostring(root, encoding="unicode", short_empty_elements=False)
302306

303307
def to_dict(self, role_key: str = "role", content_key: str = "content") -> dict[str, Any]:
304308
"""Serialize the ChatMessageContent to a dictionary.

‎python/tests/unit/contents/test_chat_message_content.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,24 @@ def test_cmc_to_prompt():
152152
assert prompt == '<message role="user"><text>Hello, world!</text></message>'
153153

154154

155+
@pytest.mark.parametrize(
156+
"encoding",
157+
[
158+
pytest.param("utf-8", id="utf8"),
159+
pytest.param("ascii", id="ascii"),
160+
pytest.param("latin-1", id="latin1"),
161+
],
162+
)
163+
def test_cmc_to_prompt_with_encoding(encoding: str):
164+
"""The encoding of the content should not change the type or shape of the prompt."""
165+
message = ChatMessageContent(role=AuthorRole.USER, content="Hello, wörld!", encoding=encoding)
166+
prompt = message.to_prompt()
167+
assert isinstance(prompt, str)
168+
assert not prompt.startswith("<?xml")
169+
assert "Hello, wörld!" in prompt
170+
assert ChatMessageContent.from_element(XML(prompt)).content == "Hello, wörld!"
171+
172+
155173
def test_cmc_from_element():
156174
element = ChatMessageContent(role=AuthorRole.USER, content="Hello, world!").to_element()
157175
message = ChatMessageContent.from_element(element)

0 commit comments

Comments
 (0)