Skip to content

Commit fc0e303

Browse files
steipetesteipete
committed
feat: add edge tts fallback provider
1 parent 6a7a1d7 commit fc0e303

11 files changed

Lines changed: 467 additions & 33 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Docs: https://docs.clawd.bot
88
- Ollama: provider discovery + docs. (#1606) Thanks @abhaymundhara. https://docs.clawd.bot/providers/ollama
99

1010
### Changes
11+
- TTS: add Edge TTS provider fallback, defaulting to keyless Edge with MP3 retry on format failures. (#1668) Thanks @steipete. https://docs.clawd.bot/tts
1112
- Docs: expand FAQ (migration, scheduling, concurrency, model recommendations, OpenAI subscription auth, Pi sizing, hackable install, docs SSL workaround).
1213
- Docs: add verbose installer troubleshooting guidance.
1314
- Docs: update Fly.io guide notes.

‎docs/channels/googlechat.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ Status: ready for DMs + spaces via Google Chat API webhooks (HTTP only).
3232
- Under **Connection settings**, select **HTTP endpoint URL**.
3333
- Under **Triggers**, select **Use a common HTTP endpoint URL for all triggers** and set it to your gateway's public URL followed by `/googlechat`.
3434
- *Tip: Run `clawdbot status` to find your gateway's public URL.*
35-
- Under **Visibility**, check **Make this Chat app available to specific people and groups in <Your Domain>**.
35+
- Under **Visibility**, check **Make this Chat app available to specific people and groups in &lt;Your Domain&gt;**.
3636
- Enter your email address (e.g. `[email protected]`) in the text box.
3737
- Click **Save** at the bottom.
3838
6) **Enable the app status**:

‎docs/tts.md‎

Lines changed: 82 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,21 +8,37 @@ read_when:
88

99
# Text-to-speech (TTS)
1010

11-
Clawdbot can convert outbound replies into audio using ElevenLabs or OpenAI.
11+
Clawdbot can convert outbound replies into audio using ElevenLabs, OpenAI, or Edge TTS.
1212
It works anywhere Clawdbot can send audio; Telegram gets a round voice-note bubble.
1313

1414
## Supported services
1515

1616
- **ElevenLabs** (primary or fallback provider)
1717
- **OpenAI** (primary or fallback provider; also used for summaries)
18+
- **Edge TTS** (primary or fallback provider; uses `node-edge-tts`, default when no API keys)
1819

19-
## Required keys
20+
### Edge TTS notes
2021

21-
At least one of:
22+
Edge TTS uses Microsoft Edge's online neural TTS service via the `node-edge-tts`
23+
library. It's a hosted service (not local), uses Microsoft’s endpoints, and does
24+
not require an API key. `node-edge-tts` exposes speech configuration options and
25+
output formats, but not all options are supported by the Edge service. citeturn2search0
26+
27+
Because Edge TTS is a public web service without a published SLA or quota, treat it
28+
as best-effort. If you need guaranteed limits and support, use OpenAI or ElevenLabs.
29+
Microsoft's Speech REST API documents a 10‑minute audio limit per request; Edge TTS
30+
does not publish limits, so assume similar or lower limits. citeturn0search3
31+
32+
## Optional keys
33+
34+
If you want OpenAI or ElevenLabs:
2235
- `ELEVENLABS_API_KEY` (or `XI_API_KEY`)
2336
- `OPENAI_API_KEY`
2437

25-
If both are configured, the selected provider is used first and the other is a fallback.
38+
Edge TTS does **not** require an API key. If no API keys are found, Clawdbot defaults
39+
to Edge TTS (unless disabled via `messages.tts.edge.enabled=false`).
40+
41+
If multiple providers are configured, the selected provider is used first and the others are fallback options.
2642
Auto-summary uses the configured `summaryModel` (or `agents.defaults.model.primary`),
2743
so that provider must also be authenticated if you enable summaries.
2844

@@ -32,12 +48,17 @@ so that provider must also be authenticated if you enable summaries.
3248
- [OpenAI Audio API reference](https://platform.openai.com/docs/api-reference/audio)
3349
- [ElevenLabs Text to Speech](https://elevenlabs.io/docs/api-reference/text-to-speech)
3450
- [ElevenLabs Authentication](https://elevenlabs.io/docs/api-reference/authentication)
51+
- [node-edge-tts](https://github.com/SchneeHertz/node-edge-tts)
52+
- [Microsoft Speech output formats](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech#audio-outputs)
3553

3654
## Is it enabled by default?
3755

3856
No. TTS is **disabled** by default. Enable it in config or with `/tts on`,
3957
which writes a local preference override.
4058

59+
Edge TTS **is** enabled by default once TTS is on, and is used automatically
60+
when no OpenAI or ElevenLabs API keys are available.
61+
4162
## Config
4263

4364
TTS config lives under `messages.tts` in `clawdbot.json`.
@@ -94,6 +115,41 @@ Full schema is in [Gateway configuration](/gateway/configuration).
94115
}
95116
```
96117

118+
### Edge TTS primary (no API key)
119+
120+
```json5
121+
{
122+
messages: {
123+
tts: {
124+
enabled: true,
125+
provider: "edge",
126+
edge: {
127+
enabled: true,
128+
voice: "en-US-MichelleNeural",
129+
lang: "en-US",
130+
outputFormat: "audio-24khz-48kbitrate-mono-mp3",
131+
rate: "+10%",
132+
pitch: "-5%"
133+
}
134+
}
135+
}
136+
}
137+
```
138+
139+
### Disable Edge TTS
140+
141+
```json5
142+
{
143+
messages: {
144+
tts: {
145+
edge: {
146+
enabled: false
147+
}
148+
}
149+
}
150+
}
151+
```
152+
97153
### Custom limits + prefs path
98154

99155
```json5
@@ -131,7 +187,9 @@ Then run:
131187

132188
- `enabled`: master toggle (default `false`; local prefs can override).
133189
- `mode`: `"final"` (default) or `"all"` (includes tool/block replies).
134-
- `provider`: `"elevenlabs"` or `"openai"` (fallback is automatic).
190+
- `provider`: `"elevenlabs"`, `"openai"`, or `"edge"` (fallback is automatic).
191+
- If `provider` is **unset**, Clawdbot prefers `openai` (if key), then `elevenlabs` (if key),
192+
otherwise `edge`.
135193
- `summaryModel`: optional cheap model for auto-summary; defaults to `agents.defaults.model.primary`.
136194
- Accepts `provider/model` or a configured model alias.
137195
- `modelOverrides`: allow the model to emit TTS directives (on by default).
@@ -147,6 +205,15 @@ Then run:
147205
- `elevenlabs.applyTextNormalization`: `auto|on|off`
148206
- `elevenlabs.languageCode`: 2-letter ISO 639-1 (e.g. `en`, `de`)
149207
- `elevenlabs.seed`: integer `0..4294967295` (best-effort determinism)
208+
- `edge.enabled`: allow Edge TTS usage (default `true`; no API key).
209+
- `edge.voice`: Edge neural voice name (e.g. `en-US-MichelleNeural`).
210+
- `edge.lang`: language code (e.g. `en-US`).
211+
- `edge.outputFormat`: Edge output format (e.g. `audio-24khz-48kbitrate-mono-mp3`).
212+
- See Microsoft Speech output formats for valid values; not all formats are supported by Edge.
213+
- `edge.rate` / `edge.pitch` / `edge.volume`: percent strings (e.g. `+10%`, `-5%`).
214+
- `edge.saveSubtitles`: write JSON subtitles alongside the audio file.
215+
- `edge.proxy`: proxy URL for Edge TTS requests.
216+
- `edge.timeoutMs`: request timeout override (ms).
150217

151218
## Model-driven overrides (default on)
152219

@@ -167,7 +234,7 @@ Here you go.
167234
```
168235

169236
Available directive keys (when enabled):
170-
- `provider` (`openai` | `elevenlabs`)
237+
- `provider` (`openai` | `elevenlabs` | `edge`)
171238
- `voice` (OpenAI voice) or `voiceId` (ElevenLabs)
172239
- `model` (OpenAI TTS model or ElevenLabs model id)
173240
- `stability`, `similarityBoost`, `style`, `speed`, `useSpeakerBoost`
@@ -225,8 +292,15 @@ These override `messages.tts.*` for that host.
225292
- 48kHz / 64kbps is a good voice-note tradeoff and required for the round bubble.
226293
- **Other channels**: MP3 (`mp3_44100_128` from ElevenLabs, `mp3` from OpenAI).
227294
- 44.1kHz / 128kbps is the default balance for speech clarity.
228-
229-
This is not configurable; Telegram expects Opus for voice-note UX.
295+
- **Edge TTS**: uses `edge.outputFormat` (default `audio-24khz-48kbitrate-mono-mp3`).
296+
- `node-edge-tts` accepts an `outputFormat`, but not all formats are available
297+
from the Edge service. citeturn2search0
298+
- Output format values follow Microsoft Speech output formats (including Ogg/WebM Opus). citeturn1search0
299+
- Telegram `sendVoice` accepts OGG/MP3/M4A; use OpenAI/ElevenLabs if you need
300+
guaranteed Opus voice notes. citeturn1search1
301+
- If the configured Edge output format fails, Clawdbot retries with MP3.
302+
303+
OpenAI/ElevenLabs formats are fixed; Telegram expects Opus for voice-note UX.
230304

231305
## Auto-TTS behavior
232306

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,7 @@
185185
"linkedom": "^0.18.12",
186186
"long": "5.3.2",
187187
"markdown-it": "^14.1.0",
188+
"node-edge-tts": "^1.2.9",
188189
"osc-progress": "^0.3.0",
189190
"pdfjs-dist": "^5.4.530",
190191
"playwright-core": "1.58.0",

‎pnpm-lock.yaml‎

Lines changed: 17 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎src/auto-reply/reply/commands-tts.ts‎

Lines changed: 31 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@ import {
77
getTtsProvider,
88
isSummarizationEnabled,
99
isTtsEnabled,
10+
isTtsProviderConfigured,
1011
resolveTtsApiKey,
1112
resolveTtsConfig,
1213
resolveTtsPrefsPath,
14+
resolveTtsProviderOrder,
1315
setLastTtsAttempt,
1416
setSummarizationEnabled,
1517
setTtsEnabled,
@@ -41,6 +43,7 @@ function ttsUsage(): ReplyPayload {
4143
"\nExamples:\n" +
4244
"/tts on\n" +
4345
"/tts provider openai\n" +
46+
"/tts provider edge\n" +
4447
"/tts limit 2000\n" +
4548
"/tts summary off\n" +
4649
"/tts audio Hello from Clawdbot",
@@ -126,33 +129,45 @@ export const handleTtsCommands: CommandHandler = async (params, allowTextCommand
126129
if (action === "provider") {
127130
const currentProvider = getTtsProvider(config, prefsPath);
128131
if (!args.trim()) {
129-
const fallback = currentProvider === "openai" ? "elevenlabs" : "openai";
132+
const fallback = resolveTtsProviderOrder(currentProvider)
133+
.slice(1)
134+
.filter((provider) => isTtsProviderConfigured(config, provider));
130135
const hasOpenAI = Boolean(resolveTtsApiKey(config, "openai"));
131136
const hasElevenLabs = Boolean(resolveTtsApiKey(config, "elevenlabs"));
137+
const hasEdge = isTtsProviderConfigured(config, "edge");
132138
return {
133139
shouldContinue: false,
134140
reply: {
135141
text:
136142
`🎙️ TTS provider\n` +
137143
`Primary: ${currentProvider}\n` +
138-
`Fallback: ${fallback}\n` +
144+
`Fallbacks: ${fallback.join(", ") || "none"}\n` +
139145
`OpenAI key: ${hasOpenAI ? "✅" : "❌"}\n` +
140146
`ElevenLabs key: ${hasElevenLabs ? "✅" : "❌"}\n` +
141-
`Usage: /tts provider openai | elevenlabs`,
147+
`Edge enabled: ${hasEdge ? "✅" : "❌"}\n` +
148+
`Usage: /tts provider openai | elevenlabs | edge`,
142149
},
143150
};
144151
}
145152

146153
const requested = args.trim().toLowerCase();
147-
if (requested !== "openai" && requested !== "elevenlabs") {
154+
if (requested !== "openai" && requested !== "elevenlabs" && requested !== "edge") {
148155
return { shouldContinue: false, reply: ttsUsage() };
149156
}
150157

151158
setTtsProvider(prefsPath, requested);
152-
const fallback = requested === "openai" ? "elevenlabs" : "openai";
159+
const fallback = resolveTtsProviderOrder(requested)
160+
.slice(1)
161+
.filter((provider) => isTtsProviderConfigured(config, provider));
153162
return {
154163
shouldContinue: false,
155-
reply: { text: `✅ TTS provider set to ${requested} (fallback: ${fallback}).` },
164+
reply: {
165+
text:
166+
`✅ TTS provider set to ${requested} (fallbacks: ${fallback.join(", ") || "none"}).` +
167+
(requested === "edge"
168+
? "\nEnable Edge TTS in config: messages.tts.edge.enabled = true."
169+
: ""),
170+
},
156171
};
157172
}
158173

@@ -199,14 +214,22 @@ export const handleTtsCommands: CommandHandler = async (params, allowTextCommand
199214
if (action === "status") {
200215
const enabled = isTtsEnabled(config, prefsPath);
201216
const provider = getTtsProvider(config, prefsPath);
202-
const hasKey = Boolean(resolveTtsApiKey(config, provider));
217+
const hasKey = isTtsProviderConfigured(config, provider);
218+
const providerStatus =
219+
provider === "edge"
220+
? hasKey
221+
? "✅ enabled"
222+
: "❌ disabled"
223+
: hasKey
224+
? "✅ key"
225+
: "❌ no key";
203226
const maxLength = getTtsMaxLength(prefsPath);
204227
const summarize = isSummarizationEnabled(prefsPath);
205228
const last = getLastTtsAttempt();
206229
const lines = [
207230
"📊 TTS status",
208231
`State: ${enabled ? "✅ enabled" : "❌ disabled"}`,
209-
`Provider: ${provider} (${hasKey ? "✅ key" : "❌ no key"})`,
232+
`Provider: ${provider} (${providerStatus})`,
210233
`Text limit: ${maxLength} chars`,
211234
`Auto-summary: ${summarize ? "on" : "off"}`,
212235
];

‎src/config/types.tts.ts‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
export type TtsProvider = "elevenlabs" | "openai";
1+
export type TtsProvider = "elevenlabs" | "openai" | "edge";
22

33
export type TtsMode = "final" | "all";
44

@@ -55,6 +55,20 @@ export type TtsConfig = {
5555
model?: string;
5656
voice?: string;
5757
};
58+
/** Microsoft Edge (node-edge-tts) configuration. */
59+
edge?: {
60+
/** Explicitly allow Edge TTS usage (no API key required). */
61+
enabled?: boolean;
62+
voice?: string;
63+
lang?: string;
64+
outputFormat?: string;
65+
pitch?: string;
66+
rate?: string;
67+
volume?: string;
68+
saveSubtitles?: boolean;
69+
proxy?: string;
70+
timeoutMs?: number;
71+
};
5872
/** Optional path for local TTS user preferences JSON. */
5973
prefsPath?: string;
6074
/** Hard cap for text sent to TTS (chars). */

‎src/config/zod-schema.core.ts‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ export const MarkdownConfigSchema = z
156156
.strict()
157157
.optional();
158158

159-
export const TtsProviderSchema = z.enum(["elevenlabs", "openai"]);
159+
export const TtsProviderSchema = z.enum(["elevenlabs", "openai", "edge"]);
160160
export const TtsModeSchema = z.enum(["final", "all"]);
161161
export const TtsConfigSchema = z
162162
.object({
@@ -207,6 +207,21 @@ export const TtsConfigSchema = z
207207
})
208208
.strict()
209209
.optional(),
210+
edge: z
211+
.object({
212+
enabled: z.boolean().optional(),
213+
voice: z.string().optional(),
214+
lang: z.string().optional(),
215+
outputFormat: z.string().optional(),
216+
pitch: z.string().optional(),
217+
rate: z.string().optional(),
218+
volume: z.string().optional(),
219+
saveSubtitles: z.boolean().optional(),
220+
proxy: z.string().optional(),
221+
timeoutMs: z.number().int().min(1000).max(120000).optional(),
222+
})
223+
.strict()
224+
.optional(),
210225
prefsPath: z.string().optional(),
211226
maxTextLength: z.number().int().min(1).optional(),
212227
timeoutMs: z.number().int().min(1000).max(120000).optional(),

0 commit comments

Comments
 (0)