Repository navigation
[fix] Resolve Gemini image MIME type instead of hard-coding image/jpeg - #9887
Conversation
VANDRANKI
left a comment
VANDRANKI
left a comment
There was a problem hiding this comment.
I read the full diff and the shared helper it wires in.
format_image_for_message in libs/agno/agno/utils/gemini.py had three branches (URL-downloaded bytes, file path, raw bytes) that all hardcoded "mime_type": "image/jpeg". This PR replaces all three with calls to agno.utils.media.resolve_image_mime_type, which I pulled and read directly since it isn't touched by this diff:
mime_type > image_format > file extension (mimetypes.guess_type) > magic bytes (get_image_type) > "image/jpeg" fallback
I checked each call site against that priority order:
- URL branch passes
image.mime_type,image.format, andimage_bytes=content_bytes(nofile_path), so a downloaded WebP with no explicit mime/format falls through to magic-byte detection. Correct. - File-path branch additionally passes
file_path=image_path, so a.pngextension resolves before bytes are even inspected. Correct. - Raw-bytes branch passes only
mime_type/format/image_bytes. Correct.
get_image_type in the same helper file checks PNG/GIF/JPEG/HEIC/WebP magic bytes and returns None for anything else, which is exactly what makes the "falls back to jpeg for unknown bytes" test pass rather than raising.
The updated test in tests/unit/models/google/test_gemini.py changes an expected mime type from image/jpeg to image/png for what looks like a real (not synthetic) test fixture, which is consistent with removing the hardcoded jpeg. The new unit tests in tests/unit/utils/test_gemini.py cover the priority order end to end (explicit mime wins over format, format wins over bytes, magic bytes for webp/jpeg, unknown bytes still default to jpeg so existing untyped-content behavior is preserved).
I did not run the suite myself, but the logic traces cleanly against the helper's own priority cascade and the tests match it. This is a low-risk, well-scoped fix: only the three previously-hardcoded lines change, and every provider using this helper already handles the returned mime type generically.
This is a community review and does not clear the merge gate.
Summary
Gemini image inputs were always labelled
image/jpeg, regardless of the actual image format.format_image_for_message()inlibs/agno/agno/utils/gemini.pyhard-coded"mime_type": "image/jpeg"in all three branches (URL, filepath, raw bytes), so a PNG/WebP/GIF/HEIC image was sent to the Gemini API declared as a JPEG. This can cause the API to reject or misinterpret the image, and it silently discards an explicitly suppliedImage.mime_type.This PR replaces the three hard-coded values with the existing helper
agno.utils.media.resolve_image_mime_type(), which is already used elsewhere in the codebase and resolves in the documented priority order: explicitmime_type>formatfield > file extension > magic bytes >image/jpegfallback. Behaviour for real JPEGs is unchanged, and the fallback keeps the previous value when nothing can be determined, so this is not a breaking change.Issue number: #9885
Type of change
Changes
libs/agno/agno/utils/gemini.pyimage_bytes=content_bytes) plus any explicitmime_type/format. The URL extension is deliberately not used here — the downloaded bytes are authoritative, and a.jpgURL that actually serves WebP is precisely the case this issue is about.mime_type/format, then the file extension, then the file's magic bytes.mime_type/format, then magic bytes.libs/agno/tests/unit/utils/test_gemini.py: 7 new regression tests covering URL/bytes/filepath resolution, explicitmime_typeprecedence, theformatfield, JPEG stability, and the unknown-bytes fallback.libs/agno/tests/unit/models/google/test_gemini.py: one existing expectation changed.test_format_messages_nests_tool_result_media_in_function_responsebuildsImage(content=b"image-bytes", mime_type="image/png")but asserted"image/jpeg"— i.e. the assertion encoded the bug being fixed here. It now asserts"image/png", matching the image the fixture declares. Flagging this explicitly since changing an existing assertion deserves a maintainer's eye.Testing
Commands actually run, in a clean
uvvenv withuv pip install -e libs/agno:mime_type, and real JPEG.image/jpeg.image/webp,image/webp,image/png,image/webp,image/jpeg.ruff format --checkon the changed files → already formatted.ruff checkon the changed files → all checks passed.mypy --config-file libs/agno/pyproject.toml libs/agno/agno/utils/gemini.py(same config./scripts/validate.shuses) → 0 errors.pytest libs/agno/tests/unit/utils/→ 488 passed, 1 skipped (excludingtest_path_safety_consistency.py, which needs theslack_sdkextra that is not installed in my environment).pytest libs/agno/tests/unit/models/google/→ 185 passed.I did not run the entire
libs/agno/tests/unitsuite: it aborts during collection on modules requiring optional vectordb/workflow extras that I could not install here. Stating this rather than claiming a green full suite. CI will cover the rest.Checklist
pyproject.tomlconfig, as./scripts/validate.shdoes)uvvenv, editable install)Duplicate and AI-Generated PR Check
Additional Notes
Related open PR. #9827 ("fix(media): standardize MIME normalization…") touches
agno/utils/media.py. It does not modifyagno/utils/gemini.py, so it does not fix this issue — the Gemini helper never called into the resolver at all. This PR is the call-site fix and builds on the helper that #9827 refines, so the two are complementary rather than competing. If #9827 lands first I'm happy to rebase.AI disclosure. This PR was generated by an AI agent (Claude), per the CONTRIBUTING requirement to disclose. Everything reported under "Testing" was actually executed and the output observed; nothing is assumed. Please review with the scrutiny that warrants, particularly the changed assertion in
test_format_messages_nests_tool_result_media_in_function_response. If you'd prefer this issue be handled by a human contributor, close it and I won't reopen.Fixes #9885