Skip to content

messages_from_google crashes with UnicodeDecodeError on inline_data image parts (Blob.data decoded as UTF-8 instead of base64-encoded) #5487

Open

Description

@BlueX888

What happened, and what did you expect?

Calling the public API inspect_ai.model.messages_from_google() on a Google GenAI Content that carries an inline_data part with real binary image bytes raises:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

(every PNG starts with \x89PNG, so this is the standard Gemini multimodal user turn). Dict input with a base64-string data hits the same path, because google-genai's Blob decodes the string to raw bytes first.

Root cause — content_from_google_parts() in src/inspect_ai/model/_google_convert.py:244-250:

elif part.inline_data and part.inline_data.mime_type and part.inline_data.data:
    content.append(
        ContentImage(
            image=as_data_uri(
                part.inline_data.mime_type,
                part.inline_data.data.decode("utf-8"),
            )
        )
    )

google.genai.types.Blob.data is typed Optional[bytes] and documented "Required. The raw bytes of the data.", so .decode("utf-8") fails on any real image, while as_data_uri() formats data:{mime_type};base64,{data} — its payload must be base64 text, not decoded raw bytes.

Two secondary effects of the same branch:

  • If the bytes happen to be UTF-8-decodable, the conversion succeeds but emits an invalid data URI (data:image/png;base64,hello-not-base64!!) — the payload is not base64.
  • The truthiness guard silently drops a part whose data is empty bytes or whose mime_type is missing, instead of converting it or erroring.

Expected: the same handling as the sibling branch for the identical Part/Blob in completion_choice_from_candidate(), src/inspect_ai/model/_providers/google.py:1975-1976:

if blob.data is not None:
    b64 = base64.b64encode(blob.data).decode()
    data_uri = f"data:{blob.mime_type};base64,{b64}"

i.e. base64-encode the raw bytes, guarding only on data is not None. That is also this module's own round-trip contract: chat_content_to_part() sends ContentImage back as Part.from_bytes(mime_type, raw_bytes).

Steps to reproduce

import asyncio
from google.genai import types as t
from inspect_ai.model import messages_from_google

PNG = b"\x89PNG\r\n\x1a\n" + b"\x00\x01\x02image-bytes\xff\xfe"

async def main():
    contents = [
        t.Content(role="user", parts=[
            t.Part(inline_data=t.Blob(mime_type="image/png", data=PNG)),
            t.Part(text="what is in this image?"),
        ]),
    ]
    try:
        msgs = await messages_from_google(contents)
        print("converted:", msgs[0].content)
    except Exception as ex:
        print(f"RAISED {type(ex).__name__}: {ex}")

asyncio.run(main())

Output (verbatim, from three independent runs — my own plus two verifier runs):

RAISED UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

The verifier runs also covered the two secondary cases, verbatim:

dict Content, b64-str data: RAISED UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
utf8-decodable bytes: uri payload NOT valid base64: 'data:image/png;base64,hello-not-base64!!'
empty data bytes -> [ContentText(..., text='kept', ...)] / missing mime_type -> [ContentText(..., text='kept', ...)]

(in the last two, the inline_data part disappears entirely — only the sibling text part survives).

Version and environment

  • inspect-ai at commit 1276f419913248bdb4c7c41a8e2274a396e97083 (current main; the src/inspect_ai/model/_google_convert.py module was added in Review tool results after execution and stop a sample on what a tool returned #5356 (c94ab26, 2026-09-11), so the regression is main-only so far, after v0.3.263)
  • Python 3.12.14, google-genai 2.24.0 (also verified with 2.22.0)
  • macOS 26.6.2 (arm64)
  • tests/model/test_google_convert.py (19 tests) has no inline_data coverage, which is why the suite stays green

Related: #3516 touched null inline_data handling in the provider path (_providers/google.py) — different file and defect. I searched open and closed issues and open PRs for messages_from_google / content_from_google_parts / inline_data and found nothing covering this.

Happy to be assigned — I can open a PR mirroring the provider branch (guard on part.inline_data is not None and part.inline_data.data is not None, then as_data_uri(mime_type, base64.b64encode(part.inline_data.data).decode()), with a default mime type when it is None) plus a regression test using real binary bytes.

AI disclosure: found and verified with an AI coding agent (Claude); the reproduction above was run by me and independently by two separate verifier runs, and I can explain and defend every line of the report.

Activity

  1. BlueX888 commented on Sep 20, 2026

    @BlueX888
    Author

    Reproduced this. Root cause: src/inspect_ai/model/_google_convert.py:249 (content_from_google_parts) decodes google-genai Blob.data, which is raw media bytes, with .decode('utf-8') and hands the result to as_data_uri(), whose payload must be base64 text.

    Minimal reproduction:

    import asyncio
    from google.genai import types as t
    from inspect_ai.model import messages_from_google
    
    PNG = b"\x89PNG\r\n\x1a\n" + b"\x00\x01\x02image-bytes\xff\xfe"
    
    asyncio.run(messages_from_google([
        t.Content(role="user", parts=[
            t.Part(inline_data=t.Blob(mime_type="image/png", data=PNG)),
            t.Part(text="what is in this image?"),
        ]),
    ]))
    

    Observed output:

    RAISED UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
    

    Fix is ready on BlueX888:fix/prep-google-convert-inline-data-utf8-decode (commit dfcd9b1d). Mirrors the provider branch in _providers/google.py: base64-encode the raw bytes, guard on inline_data/data being not None, and default a missing mime_type to image/png instead of dropping the part.
    Verified with: pytest tests/model/test_google_convert.py -q (22 passed; also 22 passed with --runtrio)

    Happy to open the PR once you confirm the approach or assign this to me.

  2. added
    acceptedIssues: maintainer-approved for external PR — the gate key
    bugSomething isn't working
    on Oct 16, 2026
  3. dragonstyle commented on Oct 16, 2026

    @dragonstyle
    Collaborator

    Accepting the narrow fix: convert inline image bytes to a base64 data URI instead of decoding them as UTF-8, using the MIME type the response provides. An unknown binary type shouldn't be assumed to be PNG.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acceptedIssues: maintainer-approved for external PR — the gate keybugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions