Skip to content

[Bug] GeminiTools.generate_video stores base64 text in Video.content, corrupting generated video artifacts #10550

Closed

Description

@BlueX888

Description

GeminiTools.generate_video builds the returned Video artifact with base64 text instead of the raw video bytes (libs/agno/agno/tools/models/gemini.py:186):

content=base64.b64encode(generated_video.video_bytes).decode("utf-8"),

agno.media.Video.content is declared as Optional[bytes] — "Raw video bytes (standardized to bytes)" (libs/agno/agno/media/media.py:530) — so Pydantic silently coerces that str into the UTF-8 bytes of the base64 text. The artifact therefore carries base64 text rather than the video, and everything downstream that calls Video.get_content_bytes() ("Get video content as raw bytes", libs/agno/agno/media/media.py:580, which returns self.content directly) gets base64 text: media offload uploads it as the video file (libs/agno/agno/utils/media_offload.py:226), and saving the artifact to disk writes a corrupt file.

Steps to Reproduce

  1. Install agno from source (verified at commit 44219f8532e2fe6ce936850455f4b9a2567e4194, agno 3.0.9) with google-genai installed.
  2. Run the script below. It stubs the google.genai client, so no API credentials or network calls are needed:
"""GeminiTools.generate_video: does Video.content hold raw bytes or base64 text?

agno.media.Video declares `content: Optional[bytes]` — "Raw video bytes
(standardized to bytes)". The sibling video tool (opencv.py) passes raw bytes.
"""
from agno.media import Video
from agno.tools.models.gemini import GeminiTools

# Fake MP4 payload
video_bytes = b"\x00\x00\x00\x18ftypmp42" + bytes(range(256))

fake_video = type(
    "FakeVideo",
    (),
    {
        "video": type("V", (), {"video_bytes": video_bytes, "mime_type": "video/mp4"})(),
    },
)()

operation = type(
    "Op",
    (),
    {
        "done": True,
        "result": type("R", (), {"generated_videos": [fake_video]})(),
    },
)()

tools = GeminiTools(api_key="test")
tools.vertexai = True  # video generation requires Vertex AI mode
tools.client = type(
    "FakeClient",
    (),
    {
        "models": type(
            "FakeModels",
            (),
            {"generate_videos": lambda self, **kw: operation},
        )(),
        "operations": type("FakeOps", (), {"get": lambda self, operation: operation})(),
    },
)()

result = tools.generate_video(agent=None, prompt="a cat")

v = result.videos[0]
print("Video.content type:", type(v.content).__name__)
print("Video.content == raw bytes? ", v.content == video_bytes)
import base64

print("Video.content == base64 text (utf-8)? ", v.content == base64.b64encode(video_bytes).decode("utf-8").encode("utf-8"))
print("get_content_bytes() == raw bytes? ", v.get_content_bytes() == video_bytes)
print("len(raw) =", len(video_bytes), "| len(content) =", len(v.content))

# What the type itself says a plain caller gets:
print("Video(content=<raw bytes>).content type:", type(Video(content=video_bytes).content).__name__)

Agent Configuration (if applicable)

# Video generation on GeminiTools requires Vertex AI mode:
agent = Agent(
    model=Gemini(...),
    tools=[GeminiTools(vertexai=True)],
)

When the model calls generate_video, the artifacts flow ToolResult.videos → function_execution_result.videos (libs/agno/agno/models/base.py:3042-3043) → run_response.videos, and media offload uploads get_content_bytes() as the video file (libs/agno/agno/utils/media_offload.py:226).

Expected Behavior

Video.content should hold the raw video bytes, per the class's own contract:

  • content: Optional[bytes] = None # Raw video bytes (standardized to bytes) — libs/agno/agno/media/media.py:530
  • Video.get_content_bytes() is documented "Get video content as raw bytes" and returns self.content directly — libs/agno/agno/media/media.py:580-582
  • Video.to_base64() (libs/agno/agno/media/media.py:634-642) base64-encodes get_content_bytes(), i.e. content is expected to be raw; pre-encoded content would be double-encoded
  • The model input path passes video.content straight to Part.from_bytes(data=video.content) (libs/agno/agno/models/google/gemini.py:1086), which expects raw bytes
  • The sibling generate_image branch in the same file passes raw bytes, with the comment "Create ImageArtifact with raw bytes (not base64 encoded)" (libs/agno/agno/tools/models/gemini.py:111-117)
  • The sibling video toolkit passes raw bytes: content=video_bytes (libs/agno/agno/tools/opencv.py:295-299); every other video toolkit passes url= — fal (libs/agno/agno/tools/fal.py:108-111), minimax (libs/agno/agno/tools/minimax.py:131-134), models_labs (libs/agno/agno/tools/models_labs.py:129), replicate (libs/agno/agno/tools/replicate.py:111), wavespeed (libs/agno/agno/tools/wavespeed.py:103), lumalab (libs/agno/agno/tools/lumalab.py:112)

The Gemini video branch is the only one that base64-encodes.

Actual Behavior

Video.content holds the UTF-8 bytes of the base64 text. For a 268-byte MP4 payload the artifact carries 360 bytes of base64 text; get_content_bytes() returns that text, and a file written from the artifact is base64 text, not a video.

Screenshots or Logs (if applicable)

Verbatim output of the script above (run 1):

[B] content type: bytes
[B] content == raw video bytes?  False
[B] content == utf-8(base64 text)?  True
[B] get_content_bytes() == raw?  False
[B] len(raw) = 268 | len(content) = 360
[C] file written from artifact == base64 text?  True / first 16 bytes on disk: b'AAAAGGZ0eXBtcDQy'

An independent second run of the same script:

Video.content type: bytes
Video.content == raw bytes?  False
Video.content == base64 text (utf-8)?  True
get_content_bytes() == raw bytes?  False
len(raw) = 268 | len(content) = 360

Environment

  • OS: macOS (arm64)
  • Agno Version: 3.0.9 (source checkout at commit 44219f8532e2fe6ce936850455f4b9a2567e4194)
  • External Dependency Versions: google-genai 2.25.0
  • Additional Environment Details: Python 3.12.14; reproduced with a stubbed google.genai client (no API calls)

Possible Solutions

Pass the raw bytes instead: content=generated_video.video_bytes (drop the base64.b64encode(...).decode("utf-8") wrapper), matching the generate_image branch in the same file and Video.content's declared contract. The mime_type/original_prompt handling stays as is. Happy to open a PR with this change.

Additional Context

  • No existing issue or PR found for this (searched the tracker for "gemini generate_video" / "generate_video base64").
  • The comment above the line (libs/agno/agno/tools/models/gemini.py:183, "Create VideoArtifact with base64 encoded content") appears stale — libs/agno/agno/tools/opencv.py:294 carries the same comment but passes raw bytes.

Activity

  1. BlueX888 commented on Sep 25, 2026

    @BlueX888
    ContributorAuthor

    Reproduced this. Root cause: libs/agno/agno/tools/models/gemini.py:186 passes base64.b64encode(video_bytes).decode("utf-8") into Video.content, which is declared Optional[bytes], so pydantic coerces it to the UTF-8 bytes of the base64 text and get_content_bytes() consumers (media offload upload, saving to disk) receive base64 text instead of the video.

    Minimal reproduction:

    import types
    from agno.tools.models.gemini import GeminiTools
    
    video_bytes = b"\x00\x00\x00\x18ftypmp42" + bytes(range(256))  # fake MP4
    op = types.SimpleNamespace(done=True, result=types.SimpleNamespace(generated_videos=[
        types.SimpleNamespace(video=types.SimpleNamespace(video_bytes=video_bytes, mime_type="video/mp4"))]))
    tools = GeminiTools(api_key="test")
    tools.vertexai = True  # video generation requires Vertex AI mode
    tools.client = types.SimpleNamespace(
        models=types.SimpleNamespace(generate_videos=lambda **kw: op),
        operations=types.SimpleNamespace(get=lambda op: op))
    v = tools.generate_video(agent=None, prompt="a cat").videos[0]
    print(v.content == video_bytes, v.get_content_bytes() == video_bytes)  # False False
    

    Observed output:

    Video.content type: bytes
    Video.content == raw bytes?  False
    Video.content == base64 text (utf-8)?  True
    get_content_bytes() == raw bytes?  False
    len(raw) = 268 | len(content) = 360
    

    Fix is ready on BlueX888:fix/prep-gemini-generate-video-stores-base64-text-as-video-content (commit 5fc2700a). Passes the raw video_bytes to Video.content, dropping the base64 wrapper — matching the generate_image branch in the same toolkit and the declared raw-bytes contract.
    Verified with: pytest tests/unit/tools/models/test_gemini.py (run from libs/agno; 11 passed, including the updated test_generate_video_success asserting content and get_content_bytes() equal the raw bytes)

    Happy to open the PR once you confirm the approach or assign this to me.

  2. sannya-singal commented on Sep 25, 2026

    @sannya-singal
    Contributor

    Resolved with #10554.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions