Repository navigation
websockets Live API starter: use realtime_input.audio/video and flush playback only on interrupted - #1433
Conversation
… playback only on interrupted media_chunks is deprecated in the Live API reference, and one call passed a dict instead of a list. The receive loop also cleared the playback queue on every turnComplete and only played parts[0]. Match the SDK-based starter: send audio and video through realtime_input, keep queued audio until the server reports an interruption, and play every inline part. Signed-off-by: YoungjuKim <[email protected]>
github-actions
Bot
added
status:awaiting review
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request updates the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Live API quickstart script to use the new schema for sending video and audio inputs, replacing the deprecated media_chunks field with video and audio fields. It also refactors the receive_audio method to improve how model interruptions and multi-part messages are handled. Feedback is provided to defensively handle cases where nested JSON fields like modelTurn or inlineData might be explicitly set to null, which could otherwise trigger an AttributeError during execution.
There was a problem hiding this comment.
Thanks @fjvbn2003 , this is a bigger fix than it looks. I tested over the raw websocket with gemini-3.8-live: the same speech clip gets no reaction via media_chunks (with or without rate=) and a correct answer via realtime_input.audio. So on main the mic is effectively dead. realtime_input.video and the interrupted handling also check out.
One line to add while you're in receive_audio: the server sends outputTranscription by default, so raw_response.decode("ascii") crashes on any non-ASCII reply (reproduce on main by typing Say hello in Japanese.). Fix:
python
response = json.loads(raw_response)
Optional: if self.audio_stream: before .close() in run(), and the bot's (… or {}) nit. Happy to re-review once the decode line is in.
… fields The server sends outputTranscription by default, so decoding each frame as ASCII raised UnicodeDecodeError on any non-ASCII reply (for example "Say hello in Japanese."). json.loads accepts the bytes directly and decodes them as UTF-8. Also treat an explicit null modelTurn/inlineData as empty, and only close the audio stream if it was opened. Assisted-by: Claude <[email protected]> Signed-off-by: YoungjuKim <[email protected]>
|
Thanks for testing it end to end! Added |
|
Follow-up on verification: I also ran |
BichengWang
left a comment
BichengWang
left a comment
There was a problem hiding this comment.
Reviewed e81009a9f05691d17062f4a4ecf5043afb3cff5a, including the earlier feedback and follow-up commits. No blocking issues found in the changed logic.
I ran eight offline checks on the actual AudioLoop class, extracted with ast to avoid importing device dependencies or initializing credentials. Stub camera, screen, and microphone inputs produce single video/audio blobs, with the microphone declaring audio/pcm;rate=16000. Receiver checks cover multiple audio parts after a text part, binary and text WebSocket frames, pending playback surviving turnComplete, and stale playback being cleared on interrupted. Unicode setup/transcription messages and the optional fields discussed earlier also pass.
The same checks against current main reproduce the old media payloads, dropped playback, and decoding failures. The full script compiles. The input fields and interruption behavior also agree with the Live API reference.
These checks use stub devices and transport; I did not exercise a real microphone, camera, speaker, or live API session.
|
Thanks @fjvbn2003, all in. Tested end to end on e81009a: voice input now works (a 5-minute spoken conversation, whereas on main the model never hears the mic because the server ignores media_chunks), screen frames sent as realtime_input.video are understood, and Say hello in Japanese. as the first message plays and prints End of turn where main crashes with UnicodeDecodeError. Good catch on the setup reply using the same decode. LGTM. |
|
Thanks a lot for testing it end to end, including the 5-minute voice session! When you have a moment, could you update your earlier "changes requested" review so the PR can be merged? |
quickstarts/websockets/Get_started_LiveAPI.pyhas drifted from the Live API reference and from the SDK version of the same script (quickstarts/Get_started_LiveAPI.py).1. It sends input with the deprecated
media_chunksfield. The Live API reference marksBidiGenerateContentRealtimeInput.mediaChunksas "DEPRECATED: Use one ofaudio,video, ortextinstead". This PR sends microphone audio asrealtime_input.audioand camera or screen frames asrealtime_input.video, the same split the SDK script uses withsend_realtime_input(audio=...)/send_realtime_input(video=...).2. The screen mode wraps a single frame incorrectly.
get_frames()sends{"media_chunks": [frame]}(a list), butget_screen()sends{"media_chunks": frame}, which puts a single object where the field expects a list. Switching tovideo, which takes one blob, removes that inconsistency.3. The audio MIME type has no sample rate. The audio is sent as
"audio/pcm". It now usesf"audio/pcm;rate={SEND_SAMPLE_RATE}", as the other Live examples do.4. Playback is flushed on every
turnCompleteand onlyparts[0]is played. The script drops queued audio whenever a turn completes, and its comment says the server sends an end of turn when you interrupt. The signal for an interruption isserverContent.interrupted, and the SDK script already flushes only on that flag. It also plays every part of a message. This PR does the same here: it plays allinlineDataparts, flushes the queue oninterrupted, and only prints onturnComplete.For context, we run a production voice-chat relay that talks to
gemini-3.8-liveover the raw WebSocket protocol. It usesrealtimeInput.audiowithaudio/pcm;rate=16000and clears client playback oninterrupted(not onturnComplete), which works as expected.The file compiles with Python 3 (
compile()). I could not run it end to end here because it needs a microphone, a camera and an API key. The diff is +24/−30 and keeps the file's CRLF line endings.(Optional, not in this PR: the SDK sends the API key in the
x-goog-api-keyheader on the Live WebSocket, and python-genai #2564 moved Live Music keys out of the URL for the same reason, since URLs end up in logs and exception messages. If header auth is supported for raw WebSocket clients, the websockets examples could show it instead of?key=. I'm happy to open an issue to discuss.)This change was prepared with the help of an AI assistant (Claude); I reviewed and verified it.
🤖 Generated with Claude Code