Skip to content

[Fix] Render image outputs in trace HTML - #1526

Open
BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-image-output-never-rendered
Open

BlueX888 wants to merge 1 commit into
guidance-ai:mainfrom
BlueX888:fix/prep-image-output-never-rendered

Conversation

@BlueX888

Copy link
Copy Markdown

Summary

trace_node_to_html never rendered ImageOutput attributes that were appended inside a role (e.g. with user(): lm += image(...)), and raised UnicodeDecodeError when an image output appeared outside a role. This PR renders image outputs in the HTML trace, inside the active role wrapper when a role is open.

Why

Two problems in trace_node_to_html (guidance/visual/_trace.py):

  1. The ImageOutput branch was an elif chained to the role-closer block (if active_role is not None: ... active_role = None, and elif isinstance(output_attr, ImageOutput):, at guidance/visual/_trace.py:90-101 on main). The closer block runs for every output attribute of a node while a role is active, so for an image appended inside a role the closer emitted an unbalanced </span> (its opening <span> was only emitted on the TextOutput path) and the elif was never reached -- the <img> tag was never emitted at all.
  2. When the branch was reached (no active role), ImageOutput.value -- a pydantic Base64Bytes field (guidance/trace/_trace.py:155-158) that holds the decoded raw image bytes -- was utf-8 decoded with .decode() and interpolated directly into the data URI, which raises UnicodeDecodeError for real image bytes (e.g. the 0x89 first byte of a PNG) and would not be valid base64 even when the decode happened to succeed. The format field was also ignored (hardcoded image/png).

Changes

  • guidance/visual/_trace.py: the role-opener emission (role <span>/<div> wrapper) now also applies to ImageOutput when a role is active, so images render inside the role wrapper and the emitted tags stay balanced; ImageOutput is handled as a sibling branch of TextOutput instead of being chained to not active_role; the data URI is built with base64.b64encode(output_attr.value) and uses output_attr.format.
  • tests/unit/test_visual.py: regression tests for an image output outside any role (utf-8-invalid PNG bytes; also format="jpeg"), and for an image output inside a role opened on a parent trace node (mirrors with user(): lm += image(...)), asserting the <img> tag is present and <span> tags are balanced.

Text-in-role rendering is unchanged (verified byte-identical output against main for both prettify_roles settings where main does not already crash).

Testing

New tests on the unpatched tree:

$ .venv/bin/python -m pytest tests/unit/test_visual.py::test_image_output_html tests/unit/test_visual.py::test_image_output_html_in_role -q
FAILED tests/unit/test_visual.py::test_image_output_html - UnicodeDecodeError...
FAILED tests/unit/test_visual.py::test_image_output_html_in_role - assert '<i...
2 failed in 0.02s

Patched tree:

$ .venv/bin/python -m pytest tests/unit/test_visual.py::test_image_output_html tests/unit/test_visual.py::test_image_output_html_in_role -q
2 passed in 0.01s

$ .venv/bin/python -m pytest tests/unit/test_visual.py -q
15 passed, 1 warning in 0.02s

$ .venv/bin/python -m pytest tests/unit -q
1639 passed, 1 skipped, 220 xfailed, 1 warning in 11.23s

$ .venv/bin/ruff format --check
150 files already formatted

$ .venv/bin/ruff check --select I,RUF022
All checks passed!

$ .venv/bin/mypy guidance
# no errors in guidance/visual/_trace.py; 119 pre-existing errors elsewhere, unchanged from main

Related issue

Related to #1077 (images not visible in output representations); that issue covers the string/ChatML representation and console display, while this PR fixes the HTML trace path (trace_node_to_html).

The ImageOutput branch of trace_node_to_html was chained to
`if active_role is not None:` as an elif, so images appended inside a
role (e.g. `with user(): lm += image(...)`) were never rendered and a
stray unbalanced </span> was emitted. When the branch was reached, the
raw image bytes held by the Base64Bytes field were utf-8 decoded for the
base64 data URI, raising UnicodeDecodeError for real images, and the
format field was ignored.

Render the image inside the active role wrapper, base64-encode the
stored bytes, and honor the format field.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant