Repository navigation
Audio/Video understanding #1556
Description
Describe the bug
> transcribe @VO.mp3
✦ I am sorry, but I cannot understand the content of the audio file you sent. If you can provide a
transcript or a summary of the audio, I would be happy to help you with it.
Client information
╭────────────────────────────────────────────────────────────────────╮
│ │
│ About Gemini CLI │
│ │
│ CLI Version 0.1.1 │
│ Git Commit 852210e1 (local modifications) │
│ Model gemini-2.5-pro │
│ Sandbox no sandbox │
│ OS darwin │
│ │
╰────────────────────────────────────────────────────────────────────╯
Additional context
The README mentions Gemini's multimodal capabilities. When using Gemini API, it supports audio/video input, but when using Gemini CLI, it refuses. I asked Gemini to diagnose itself, here's what it has to say:
✦ From my perspective as the model operating within the Gemini CLI, my core intelligence is indeed multi-modal and capable of processing various data types, including audio.
However, I am dependent on the tools provided to me within this specific command-line environment to interact with your local files. The tool I use for accessing file content,
read_file, does not currently support processing audio formats like .mp3.When you referenced
@VO.mp3, theread_filetool attempted to access it and returned the message: "Cannot display content of binary file". I received only that error message, not the audio data itself. Therefore, the limitation is not in my analytical capability but in the data ingestion mechanism of the tools available in my current environment.
Activity
allenhutchison commentedon Jun 25, 2025
Right now, read_file supports text, images (PNG, JPG, GIF, WEBP, SVG, BMP), and PDF files.
Gable-github commentedon Jun 25, 2025
hi, I'm curious about the future plans for Gemini CLI, particularly around expanding its multimodal capabilities. Is the plan to increase support for additional filetypes incrementally, adding them by PRs one by one?
I’d love to contribute to the project, maybe by helping implement support for a new filetype or a similar feature @allenhutchison
fhinkel commentedon Jun 25, 2025
Gemini CLI can handle audio if the video lives on Youtube for example. Here's my coworker Smitha showing an example https://www.linkedin.com/posts/smithakolan_i-just-used-the-new-gemini-cli-to-set-up-activity-7343679677594484736-LbJb?utm_source=share&utm_medium=member_desktop&rcm=ACoAABfA4eIBuSqMEo5g7a3wzq_QqP_7fOeAnHA
Gable-github commentedon Jun 25, 2025
@fhinkel i think that is a great workaround. but correct me if im wrong, the video is just gemini doing 2 web fetches:
but there's no audio processing it seems
markmcd commentedon Jun 26, 2025
@Gable-github is correct, that example does not actually use youtube, and only works if the content is available elsewhere on the internet. This FR is for integrated support.
jacob314 commentedon Jun 27, 2025
This might be a good first issue. when I added this functionality I fairly arbitrarily restricted the formats to image and pdf. You should likely be able to extend this by adding support to the detectFileType method in fileUtils.ts and then the appropriate support in read-file.ts and then update the uses of detectFileType to handle video and audio file types. In particular look at processSingleFileContent in fileUtils.ts and update that appropriately for your file types likely just matching what happens for 'image' and 'pdf'
Gable-github commentedon Jun 28, 2025
I would like to work on this issue. could someone with permissions please assign it to me, or should I go ahead and start a draft PR (seems to be the case for the other issues)? thanks!
ShubhamSachdeva311205 commentedon Jun 29, 2025
Hey Gabe:)
I was actually hoping to work on this issue too. Totally happy to collaborate if you're open to that, or if you’ve already started something, I’d love to follow along and help however I can. Let me know what works best!
m-bain commentedon Jul 6, 2025
Need to update tool documentation, currently gemini-cli thinks it can't read videos
#3380
lukehutch commentedon Feb 13, 2026
I would also love to see Gemini CLI be able to support voice input directly (type
/speak, and the mic device is opened, so that you can speak into the console).I'm also using Gemini CLI as a backend (as an agent) in a larger system, and I need voice input in that system. It would be awesome if there were some way (via a commandline call or a Gemini CLI API call of some sort) to have Gemini CLI serve as a "voice server" for voice recognition from another process.