feat(google): add Gemini TTS provider by barronlroth · Pull Request #67515 · openclaw/openclaw

barronlroth · 2026-04-16T05:07:42Z

Summary

Add a bundled Google speech provider for Gemini API TTS using the existing google provider id.
Call Gemini generateContent with responseModalities: ["AUDIO"], preserve Gemini audio tags in the transcript, wrap 24 kHz PCM as WAV for normal TTS, and return PCM directly for Talk/telephony.
Reuse API-key-only auth surfaces: messages.tts.providers.google.apiKey, models.providers.google.apiKey, GEMINI_API_KEY, and GOOGLE_API_KEY.
Register contracts.speechProviders: ["google"] and update docs/plan for config, output format, Cloud-vs-Gemini key semantics, and native voice-note limitations.

Closes #67458

Security

Adds a new outbound TTS network path to the Gemini API through the existing Google Generative AI HTTP helper.
Keeps the trusted base URL restriction on https://generativelanguage.googleapis.com and does not add Cloud Text-to-Speech, OAuth, or auth-profile reuse in this phase.
Handles API keys only through existing secret/config/env surfaces and sends them as Gemini API x-goog-api-key headers.

Validation

pnpm test:extension google
pnpm test:contracts:plugins
pnpm exec oxlint --type-aware -- extensions/google/index.ts extensions/google/speech-provider.ts extensions/google/speech-provider.test.ts extensions/google/plugin-registration.contract.test.ts extensions/google/test-api.ts test/helpers/plugins/plugin-registration-contract-cases.ts
git diff --check
pnpm tsgo currently fails on pre-existing Telegram AbortSignal type mismatches in extensions/telegram/src/bot.ts:193 and extensions/telegram/src/bot.ts:195; no Google TTS type errors were reported before that blocker.

AI Assistance

This implementation and validation were assisted by Codex.

greptile-apps · 2026-04-16T05:16:50Z

Greptile Summary

This PR adds a google speech provider to the bundled Google plugin, wiring up Gemini API TTS (gemini-3.1-flash-tts-preview) through the existing resolveGoogleGenerativeAiHttpRequestConfig helper. Auth key resolution, PCM-to-WAV wrapping, telephony PCM passthrough, voice-directive parsing, and registration contract updates are all present. The implementation follows the same patterns as existing providers and reuses the trusted-base-URL enforcement already in api.ts.

Two minor items worth a look before merge: the plan.md working document was committed to the repo root (open TODOs and follow-up questions suggest it's a scratch file), and an unrelated type narrowing on GoogleMediaUnderstandingProvider in index.ts (removing the MediaUnderstandingProvider & intersection) slipped in without explanation.

Confidence Score: 5/5

Safe to merge; no logic or security defects found in the new speech provider path.

All remaining findings are P2: a planning document committed to the repo root, and an unrelated type-narrowing change. Neither affects runtime correctness or security. The core implementation — key resolution, WAV wrapping, URL validation, and registration — is correct and consistent with existing providers.

plan.md (should be removed); extensions/google/index.ts (unexplained type change)

Comments Outside Diff (1)

extensions/google/index.ts, line 15-20 (link)

Unrelated type narrowing: intersection removed

The MediaUnderstandingProvider & intersection was dropped here, changing the type from an intersection (all MediaUnderstandingProvider members plus non-null assertions on the four methods) to a plain object type with only those four methods. Since GoogleMediaUnderstandingProvider is only used internally for the cast in loadGoogleRequiredMediaUnderstandingProvider, this doesn't affect runtime, but it's a quiet structural change unrelated to the speech provider feature. If it was needed to resolve a type error, a brief comment explaining why would help future readers.

Prompt To Fix With AI

This is a comment left during a code review.
Path: extensions/google/index.ts
Line: 15-20

Comment:
**Unrelated type narrowing: intersection removed**

The `MediaUnderstandingProvider &` intersection was dropped here, changing the type from an intersection (all `MediaUnderstandingProvider` members plus non-null assertions on the four methods) to a plain object type with only those four methods. Since `GoogleMediaUnderstandingProvider` is only used internally for the cast in `loadGoogleRequiredMediaUnderstandingProvider`, this doesn't affect runtime, but it's a quiet structural change unrelated to the speech provider feature. If it was needed to resolve a type error, a brief comment explaining why would help future readers.

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: plan.md
Line: 1

Comment:
**Planning artifact committed to repo root**

`plan.md` is a working implementation document (with open TODO items and follow-up questions) that was added to the repository root. Planning/scratch files like this are not typically meant to be permanent parts of the codebase — the repo root is not the right home for them. If this should be preserved for historical context, consider moving it to a branch-local scratch location or removing it before merge.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: extensions/google/index.ts
Line: 15-20

Comment:
**Unrelated type narrowing: intersection removed**

The `MediaUnderstandingProvider &` intersection was dropped here, changing the type from an intersection (all `MediaUnderstandingProvider` members plus non-null assertions on the four methods) to a plain object type with only those four methods. Since `GoogleMediaUnderstandingProvider` is only used internally for the cast in `loadGoogleRequiredMediaUnderstandingProvider`, this doesn't affect runtime, but it's a quiet structural change unrelated to the speech provider feature. If it was needed to resolve a type error, a brief comment explaining why would help future readers.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "Add Google Gemini TTS provider" | Re-trigger Greptile}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7db083a1ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…roth

…roth)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da2c085c9e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…barronlroth

…barronlroth)

…rronlroth

…rronlroth)

obviyus

Verified the Google Gemini TTS registration and synthesis path, including targeted coverage for provider registration, WAV/PCM output, and config-backed provider selection.

Maintainer follow-up: added the Unreleased changelog entry, fixed auto-selection so models.providers.google.apiKey counts during primary TTS provider selection, and narrowed Google-specific directive aliases to avoid shadowing by higher-priority providers.

Local gate: pnpm test src/plugins/contracts/tts.config.contract.test.ts extensions/google/speech-provider.test.ts extensions/google/plugin-registration.contract.test.ts and pnpm check.

obviyus · 2026-04-16T06:24:53Z

Landed on main.

Landed source commit: 8f1d391
Merge commit: bf59917

Thanks @barronlroth.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f1d391371

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-16T06:33:36Z

+  return normalizeResolvedSecretInputString({
+    value: cfg?.models?.providers?.google?.apiKey,
+    path: "models.providers.google.apiKey",


Handle SecretRef model-provider keys in Google TTS auth fallback

resolveGoogleTtsModelProviderApiKey reads models.providers.google.apiKey with normalizeResolvedSecretInputString, which throws on unresolved SecretRef inputs instead of treating them as not-yet-resolved values. Since both isConfigured and synthesis call this path, a config that stores the Google model-provider key as a SecretRef will make Google TTS fail during provider selection/runtime checks before any request is sent, even though the fallback path is documented as supported.

Useful? React with 👍 / 👎.

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

…roth

…roth) * Add Google Gemini TTS provider * Remove committed planning artifact * Explain Google media provider type shape * google: distill Gemini TTS provider * fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronlroth) * fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @barronlroth) * fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @barronlroth) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

openclaw-barnacle Bot added docs Improvements or additions to documentation size: L labels Apr 16, 2026

barronlroth changed the title ~~[codex] Add Google Gemini TTS provider~~ [Feature] Add Google Gemini TTS provider Apr 16, 2026

barronlroth changed the title ~~[Feature] Add Google Gemini TTS provider~~ feat(google): add Gemini TTS provider Apr 16, 2026

barronlroth marked this pull request as ready for review April 16, 2026 05:13

greptile-apps Bot reviewed Apr 16, 2026

View reviewed changes

Comment thread plan.md Outdated

barronlroth mentioned this pull request Apr 16, 2026

[Feature]: Add provider-specific TTS prompt hints #67539

Closed

chatgpt-codex-connector Bot reviewed Apr 16, 2026

View reviewed changes

Comment thread extensions/google/speech-provider.ts

obviyus self-assigned this Apr 16, 2026

barronlroth and others added 5 commits April 16, 2026 11:23

Add Google Gemini TTS provider

9ae47f2

Remove committed planning artifact

fcbaa65

Explain Google media provider type shape

983547b

google: distill Gemini TTS provider

8b48e60

fix: add Google Gemini TTS provider (openclaw#67515) (thanks @barronl…

da2c085

…roth)

obviyus force-pushed the codex/add-gemini-tts-provider branch from 7db083a to da2c085 Compare April 16, 2026 05:54

chatgpt-codex-connector Bot reviewed Apr 16, 2026

View reviewed changes

Comment thread extensions/google/speech-provider.ts Outdated

Comment thread extensions/google/speech-provider.ts Outdated

obviyus added 2 commits April 16, 2026 11:45

fix: honor cfg-backed Google TTS selection (openclaw#67515) (thanks @…

ce3add2

…barronlroth)

fix: narrow Google TTS directive aliases (openclaw#67515) (thanks @ba…

8f1d391

…rronlroth)

obviyus approved these changes Apr 16, 2026

View reviewed changes

obviyus merged commit bf59917 into openclaw:main Apr 16, 2026
8 checks passed

chatgpt-codex-connector Bot reviewed Apr 16, 2026

View reviewed changes

github-actions Bot mentioned this pull request Apr 16, 2026

📡 Upstream Digest — 2026-04-16 07:21 UTC curtismercier/openclaw-mods#588

Open

barronlroth deleted the codex/add-gemini-tts-provider branch April 17, 2026 23:42

cshape mentioned this pull request Apr 25, 2026

feat(tts): add Inworld as built-in speech provider #55972

Merged

5 tasks

clawsweeper Bot mentioned this pull request Jun 25, 2026

fix(google): bound JSON response reads to prevent OOM #96324

Closed

This was referenced Jul 14, 2026

fix(google): reject malformed provider media base64 #106969

Open

[Feature]: Add FlowSpeech text-to-speech provider #108128

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(google): add Gemini TTS provider#67515

feat(google): add Gemini TTS provider#67515
obviyus merged 7 commits into
openclaw:mainfrom
barronlroth:codex/add-gemini-tts-provider

barronlroth commented Apr 16, 2026

Uh oh!

greptile-apps Bot commented Apr 16, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

obviyus left a comment

Uh oh!

Uh oh!

obviyus commented Apr 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

barronlroth commented Apr 16, 2026

Summary

Security

Validation

AI Assistance

Uh oh!

greptile-apps Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Comments Outside Diff (1)

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

obviyus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

obviyus commented Apr 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Apr 16, 2026 •

edited

Loading