Skip to content

[fix] Nest Gemini tool-result media in function responses - #9647

Merged
sannya-singal merged 7 commits into
agno-agi:mainfrom
green3sf:fix/gemini-multimodal-tool-results
Aug 26, 2026
Merged

[fix] Nest Gemini tool-result media in function responses#9647
sannya-singal merged 7 commits into
agno-agi:mainfrom
green3sf:fix/gemini-multimodal-tool-results

Conversation

@green3sf

@green3sf green3sf commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Encode supported Gemini tool-result images and document blobs inside FunctionResponse.parts instead of mixing function_response and inline_data as sibling parts.
  • Enable the nested representation only for Gemini 3 and later models, preserving the existing sibling representation for legacy models and unsupported media types.
  • Allow URI-backed response media on Vertex AI only when its MIME type is supported by multimodal function responses.
  • Add regression coverage for nested inline media, legacy-model fallback, unsupported-media fallback, and Vertex URI filtering.

(If applicable, issue number: #9630)

Fixes #9630

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Improvement
  • Model update
  • Other:

Checklist

  • Code complies with style guidelines
  • Ran format/validation scripts (./scripts/format.sh and ./scripts/validate.sh)
  • Self-review completed
  • Documentation updated (comments, docstrings)
  • Examples and guides: Relevant cookbook examples have been included or updated (if applicable)
  • Tested in clean environment
  • Tests added/updated (if applicable)

Duplicate and AI-Generated PR Check

  • I have searched existing open pull requests and confirmed that no other PR already addresses this issue
  • If a similar PR exists, I have explained below why this PR is a better approach
  • Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.)

Additional Notes

Codex assisted with implementation and test preparation. I reviewed the final diff against the latest main, verified the generated payload shape against the official multimodal function-response contract, and ran the following checks:

  • pytest -q libs/agno/tests/unit/models/google: 185 passed with google-genai 2.19.0
  • Four focused multimodal regression tests with google-genai 1.52.0 and 2.19.0: passed
  • ruff format --check and ruff check on the changed files: passed
  • ./scripts/format.sh: passed
  • ./scripts/validate.sh: Ruff and cookbook checks passed. Mypy still reports existing diagnostics in the Google SDK compatibility/Interactions code and pre-existing Gemini lines, plus missing typer in the minimal test environment; it reported no diagnostics in the new compatibility guard or tests.
  • git diff --check: passed

@green3sf
green3sf marked this pull request as ready for review August 20, 2026 18:49
@green3sf
green3sf requested a review from a team as a code owner August 20, 2026 18:49
@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR updates Gemini tool-result serialization so supported media is nested within function responses for Gemini 3 and later while preserving sibling-part fallback behavior for legacy models and unsupported media.

  • Adds model-version and MIME-type capability gates for multimodal function responses.
  • Supports nested inline data and Vertex URI media.
  • Adds regression tests for nested media and fallback paths.

Confidence Score: 5/5

The PR appears safe to merge because no concrete changed-code failure remains after checking the supported SDK range, model identifiers, and media fallback paths.

The new serialization is restricted to compatible Gemini models and supported MIME types, while all other tool-result media continues through the existing sibling-part representation and the principal branches have regression coverage.

Important Files Changed
Filename Overview
libs/agno/agno/models/google/gemini.py Adds capability-gated conversion of supported tool-result media into nested function-response parts while retaining sibling fallbacks.
libs/agno/tests/unit/models/google/test_gemini.py Adds focused coverage for nested inline media, legacy and unsupported-media fallbacks, and filtered Vertex URI media.

Flowchart

 
%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Tool-result message] --> B[Format attached media]
    B --> C{Gemini 3 or later?}
    C -- No --> F[Emit media as sibling parts]
    C -- Yes --> D{Supported MIME type?}
    D -- No --> F
    D -- Yes --> E{Inline data or Vertex URI?}
    E -- Supported --> G[Nest media in FunctionResponse.parts]
    E -- Unsupported --> F
    G --> H[Emit tool message]
    F --> H
Loading

Reviews (1): Last reviewed commit: "fix: guard Gemini multimodal response co..." | Re-trigger Greptile

This was referenced Aug 26, 2026
Closed
Closed
@sannya-singal sannya-singal self-assigned this Aug 26, 2026
@sannya-singal
sannya-singal merged commit 6259a85 into agno-agi:main Aug 26, 2026
10 checks passed
@ysolanky ysolanky mentioned this pull request Aug 26, 2026
Merged
ysolanky added a commit that referenced this pull request Aug 26, 2026
@ysolanky
chore: release 3.0.1 (#9794)
81b2784 
# Changelog

## **Improvements**

- **Faster agent runs with many tools**: Tool schemas are now derived
once and cached across runs, cutting per-run overhead for agents that
carry large toolkits. (#9771)
- **Faster long conversations**: Session history is loaded incrementally
per turn, so response time stays flat as a conversation grows instead of
scaling with its length. (#9775)
- **PubMed request timeout**: `PubmedTools` now accepts a `timeout` and
passes it to both NCBI E-utilities requests, so a stalled PubMed
response cannot hold a tool call indefinitely. (#9463)

## **Bug Fixes**

- **Strict-mode tool schemas without `properties`**:
`Function.process_schema_for_strict` no longer raises `KeyError` on
schemas that omit `properties` (for example MCP server schemas
registered verbatim). (#9578, fixes #9409)
- **Gemini tool-result media**: Images and document blobs returned from
tools are now nested inside `FunctionResponse.parts` for Gemini 3 and
later models instead of being emitted as sibling `inline_data` parts;
legacy models keep the previous representation. URI-backed response
media on Vertex AI is only sent when the MIME type is supported. (#9647)
- **Location lookup on core-only installs**: `agno.utils.location` used
the undeclared `requests` package, so `add_location_to_context=True`
raised `ModuleNotFoundError` without extras installed. It now uses
`httpx`. (#9793, fixes #9772)
- **`ScheduleManager` cleanup**: `close()` tolerates a partially
constructed manager (for example after a failed `deepcopy`), so garbage
collection no longer raises `AttributeError` on a missing `_pool`.
(#9792)

## **Cookbooks**

- Added a deterministic side-effect approval flow example. (#9790)
- Added DeepKeep AI Firewall guardrails example. (#9784)
- Documented AgentOS team run states. (#9768)
- Added a Peer Cash MCP agent example. (#9562)
- Fixed stale cookbook paths left over from the directory renumber and
in the MCP examples. (#9789, #9783)
@RaphalRiad RaphalRiad mentioned this pull request Sep 29, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sannya-singal sannya-singal sannya-singal approved these changes

Assignees

@sannya-singal sannya-singal

Labels

first-time-contributor

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Gemini 3.7: 400 INVALID_ARGUMENT when ToolResult contains inline_data (function_response and blob in same message)

2 participants

@green3sf @sannya-singal