feat(xai): add native streaming TTS by TheAngryPit · Pull Request #103993 · openclaw/openclaw

TheAngryPit · 2026-07-10T23:35:01Z

What Problem This Solves

Closes #99660 by adding native xAI WebSocket streaming TTS through the existing speech-provider contract. Buffered xAI TTS, voice discovery, authentication, and telephony remain provider-owned.

Why This Change Was Made

Uses xAI's canonical wss://api.x.ai/v1/tts protocol with ordered text.delta / text.done input and audio.delta / audio.done output.
Keeps bearer credentials on the canonical xAI host by rejecting altered ports, credentials, paths, queries, and fragments.
Bounds frames and total decoded audio, refreshes idle timeouts on audio, preserves Unicode chunk boundaries, and makes cancellation/release idempotent.
Keeps output policy in the xAI provider: voice-note targets use self-describing MP3 for both streaming and buffered fallback; audio-file targets retain configured MP3, WAV, PCM, mu-law, or A-law.

User Impact

xAI TTS can now stream through registered speech-provider callers. Discord receives portable MP3 voice-note bytes on both the stream and fallback paths; no xAI-specific codec policy remains in Discord. Telephony stays PCM. Audio-file callers keep all supported configured codecs.

Evidence

Exact head: 156879a36e6a449b0290d1c41416800e98f9e55a

Focused Blacksmith Testbox proof
  corepack pnpm test extensions/xai/tts.test.ts \
    extensions/xai/speech-provider.test.ts \
    packages/speech-core/src/tts.test.ts \
    extensions/discord/src/voice/manager.e2e.test.ts
Result: 4 files, 227 tests passed
Lease: tbx_01kxdr5dr7w8wyvqcbqhgg5qd0
Run: https://github.com/openclaw/openclaw/actions/runs/29251061965

Broad Blacksmith Testbox proof
  corepack pnpm check:changed && corepack pnpm build
Result: passed; formatting, dependency guard, SDK surface, extension/test types,
lint, database-first policy, import-cycle checks, build, UI, and SDK exports green

Authenticated live xAI proof
  OPENCLAW_LIVE_TEST=1 node scripts/test-live.mjs \
    extensions/xai/xai.live.test.ts \
    -t "synthesizes TTS through the registered speech provider"
Result: 1 passed, 9 unrelated live cases skipped
Covered real voice catalog, buffered TTS, WebSocket audio bytes, and telephony.

git diff --check
Result: passed

Scope

Seven files: xAI provider/runtime/tests plus xAI and TTS docs.
No Discord code changes.
No dependency or lockfile changes; [email protected] already belongs to the xAI plugin runtime on current main.
No realtime-voice changes, gateway changes, new config, or sample-rate/bit-rate surface.

No API keys, tokens, credentials, private paths, raw prompts, or raw agent logs are included.

Agent Transcript

Sanitized engineering worklog

Audited the original streaming implementation against the provider owner boundary, registered callers, Discord fallback path, tests, current main, ws runtime behavior, and current xAI protocol documentation.
Removed provider-specific codec policy from Discord and made xAI own voice-note versus audio-file output selection.
Added regression coverage for streaming and buffered voice-note fallback plus every supported audio-file codec.
Proved the exact head with focused and broad Testbox runs and an authenticated live registered-provider smoke.

Strongest safe truth: the exact head is live-proven for native xAI voice discovery, buffered TTS, WebSocket streaming audio, and telephony. It is not claimed as a live Discord upload proof.

github-actions · 2026-07-10T23:35:32Z

Dependency graph guard cleared

This PR no longer has blocked dependency graph changes. A future dependency graph change requires a fresh /allow-dependencies-change comment after the guard blocks that new head SHA.

Current SHA: 156879a36e6a449b0290d1c41416800e98f9e55a

clawsweeper · 2026-07-10T23:50:35Z

Codex review: needs maintainer review before merge. Reviewed July 11, 2026, 8:59 AM ET / 12:59 UTC.

Summary
The PR adds native xAI WebSocket streaming TTS through the existing speech-provider contract, applies MP3 only to Discord streaming playback, and adds provider tests, live coverage, documentation, and a plugin-local ws runtime dependency.

PR surface: Source +404, Tests +554, Docs +8, Config 0, Other 0. Total +966 across 11 files.

Reproducibility: not applicable. This PR adds a new provider capability rather than fixing broken established behavior, and its exact-head live run validates the proposed streaming path.

Review metrics: 3 noteworthy metrics.

Runtime Dependency Ownership: 1 reclassified, 0 new resolved versions. The existing pinned ws version becomes a production plugin dependency, which is architecturally appropriate but requires exact-head authorization.
Streaming Codec Surface: 5 provider formats retained, 1 Discord stream format forced. The provider preserves its established codec contract while Discord narrows only streamed playback to MP3 and leaves buffered fallback settings intact.
Streaming Lifecycle Paths: 5 covered: completion, server error, timeout, cancellation, early close. The implementation and tests cover the main settlement modes that could otherwise leak sockets, hang playback, or suppress errors.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #99660
Summary: This PR is the proof-positive active implementation candidate for the canonical native xAI streaming TTS request; the two earlier implementation attempts closed unmerged and are not viable canonical landing paths.

Members:

canonical: feat(xai): add native streaming TTS WebSocket support #99660 - The issue specifies the same native endpoint, provider hook, codec scope, security restriction, and proof target.
superseded: feat(xai): stream TTS over native websocket #99868 - This earlier implementation closed unmerged and lacked live provider proof.
superseded: feat(xai): stream TTS over native websocket #99883 - This earlier resubmission closed unmerged and lacked live provider proof.
adjacent_distinct: Fix voice-call streaming provider resolution #97170 - That merged PR concerns xAI realtime transcription provider resolution, which is a separate speech capability and protocol path.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] Merging changes the packaged xAI plugin's production dependency graph; the exact reviewed head cannot land until a repository administrator or security owner explicitly authorizes [email protected] as a runtime dependency.

Maintainer options:

Authorize the exact-head dependency (recommended)
Approve [email protected] as the xAI plugin's runtime WebSocket client after confirming the importer-only graph change.
Replace only with proven parity
Use another client only if endpoint pinning, bearer headers, payload limits, timeout refresh, cancellation, release, and supported-runtime behavior remain equivalent.
Pause native streaming
Do not merge native xAI streaming if production dependency ownership is not accepted.

Next step before merge

[P2] The only remaining merge action is a human exact-head dependency authorization; no concrete automated code repair is indicated.

Maintainer decision needed

Question: Should [email protected] be authorized as the xAI plugin's production WebSocket client at head a1b1595?
Rationale: The implementation, dependency version, and plugin ownership are appropriate, but repository policy reserves external dependency-graph authorization for a repository administrator or security owner.
Likely owner: steipete — This person has the strongest merged-history connection to both the current xAI TTS surface and the Discord streaming TTS caller, although final dependency authorization may require a separate security owner.
Options:
- Authorize plugin-local ws (recommended): Approve the existing pinned package and importer-only lockfile reclassification for this exact head.
- Require equivalent runtime parity: Remove ws only if another supported client proves equivalent authorization headers, payload bounds, lifecycle behavior, and Node/Bun compatibility.
- Defer native streaming: Keep the feature unmerged if maintainers do not want to own this production dependency.

Security
Needs attention: Endpoint pinning, bearer handling, payload bounds, and lockfile scope are sound, but exact-head authorization of the production dependency remains required.

Review details

Best possible solution:

Authorize the exact-head plugin-local ws dependency and merge the current implementation, preserving vendor protocol handling inside the xAI plugin and keeping Discord's MP3 policy limited to streaming playback.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds a new provider capability rather than fixing broken established behavior, and its exact-head live run validates the proposed streaming path.

Is this the best way to solve the issue?

Yes. The patch reuses the canonical speech-provider streaming seam, keeps xAI-specific protocol logic in the xAI plugin, and confines Discord's MP3 requirement to streaming playback without altering buffered fallback.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a66d2959e956.

Label changes

Label justifications:

P2: This is a bounded provider capability improvement with strong proof and limited blast radius, not an urgent regression.
merge-risk: 🚨 compatibility: The packaged xAI plugin gains a production runtime dependency, changing installation and upgrade requirements even though the resolved version was already present.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body contains exact-head live xAI proof that drained non-empty streamed audio and asserted returned format metadata, byte count, and release behavior.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body contains exact-head live xAI proof that drained non-empty streamed audio and asserted returned format metadata, byte count, and release behavior.

Evidence reviewed

PR surface:

Source +404, Tests +554, Docs +8, Config 0, Other 0. Total +966 across 11 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	3	407	3	+404
Tests	4	554	0	+554
Docs	2	13	5	+8
Config	1	3	3	0
Generated	0	0	0	0
Other	1	3	3	0
Total	11	980	14	+966

Security concerns:

[medium] Authorize the xAI runtime dependency — extensions/xai/package.json:8
The plugin now imports ws in production and moves the already-pinned package into runtime dependencies; repository policy requires administrator or security approval for this exact head.
Confidence: 0.99

What I checked:

Existing extension seam: Speech core already prepares provider settings, invokes streamSynthesize, and forwards the resulting stream, metadata, and release callback, so the feature does not require a parallel core API. (packages/speech-core/src/tts.ts:1640, a66d2959e956)
Provider implementation: The xAI plugin implements the native streaming protocol locally, pins the canonical endpoint, bounds frames and decoded audio, refreshes idle timeouts, and handles completion, errors, cancellation, and repeated release. (extensions/xai/tts.ts:176, a1b159567da3)
Provider contract integration: streamSynthesize mirrors buffered synthesis for credential, voice, language, speed, format, timeout, and generated-audio limit resolution while returning the established stream result shape. (extensions/xai/speech-provider.ts:271, a1b159567da3)
Buffered fallback compatibility: Discord injects responseFormat: "mp3" only into the streaming overrides and continues passing the original configuration to buffered fallback, including persona-bound settings. (extensions/discord/src/voice/tts.ts:126, a1b159567da3)
Regression coverage: The exact head covers endpoint validation, frame ordering and limits, surrogate-pair handling, payload and cumulative audio bounds, timeout refresh, early close, server errors, cancellation, idempotent release, provider integration, and Discord codec fallback behavior. (extensions/xai/tts.test.ts:1, a1b159567da3)
Real behavior proof: The PR evidence records an exact-head credentialed live xAI run that drained a real stream and asserted non-empty audio bytes, MP3 metadata, file extension, and release behavior. (extensions/xai/xai.live.test.ts:267, a1b159567da3)

Likely related people:

steipete: Recent merged history includes the current xAI voice-discovery work and the original Discord streaming TTS caller, making this the strongest routing candidate for provider/caller fit. (role: feature owner and recent area contributor; confidence: high; commits: 7f5dce8bbf1b, 5aefe6abd6be, 77d9ac30bb8d; files: extensions/xai/tts.ts, extensions/xai/speech-provider.ts, extensions/discord/src/voice/tts.ts)
vincentkoc: Recent merged work added generated-speech byte limits and xAI telephony override handling, both directly adjacent to this PR's limits and provider-resolution behavior. (role: adjacent speech safety contributor; confidence: medium; commits: 4438be7f0592, 361737d1f1cb, 84e5327720b4; files: extensions/xai/tts.ts, extensions/xai/speech-provider.ts, packages/speech-core/src/tts.ts)
Jaaneek: Merged xAI OAuth and user-agent attribution work owns credential and request metadata behavior reused by the new authenticated WebSocket path. (role: adjacent xAI auth contributor; confidence: medium; commits: 5f1df99a9c23; files: extensions/xai/tts.ts, extensions/xai/speech-provider.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (3 earlier review cycles)

reviewed 2026-07-10T23:48:57.353Z sha cde93da :: needs changes before merge. :: [P1] Preserve fallback for unsupported streaming codecs
reviewed 2026-07-11T11:02:01.540Z sha f08d1fd :: needs changes before merge. :: [P3] Format the new xAI streaming implementation
reviewed 2026-07-11T12:47:14.808Z sha a1b1595 :: needs maintainer review before merge. :: none

Expose streamSynthesize on the bundled xAI speech provider so Talk and Discord streaming playback can use xAI alongside batch REST synthesis. Adds ws runtime dependency, byte-cap enforcement, and provider tests.

Restrict streaming WS to api.x.ai, decode WebSocket payloads explicitly, refresh idle timeout on audio.delta, add lifecycle/security tests, and document the official xAI streaming contract plus non-realtime-voice scope.

TheAngryPit · 2026-07-11T10:46:01Z

@clawsweeper re-review

clawsweeper · 2026-07-11T10:46:04Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29150084793
Updated: 2026-07-11T11:03:40.037Z

steipete · 2026-07-13T13:08:09Z

Land-ready at exact head 156879a36e6a449b0290d1c41416800e98f9e55a.

Repair completed:

Moved all xAI output-format policy into the xAI provider.
Voice-note streaming and buffered fallback both produce MP3.
Audio-file callers retain MP3, WAV, PCM, mu-law, and A-law choices; telephony remains PCM.
Removed the xAI-specific Discord diff and obsolete dependency changes.

Proof:

Blacksmith Testbox tbx_01kxdr5dr7w8wyvqcbqhgg5qd0: 4 focused files, 227 tests passed (run).
Same exact-head Testbox: corepack pnpm check:changed && corepack pnpm build passed.
Authenticated live xAI registered-provider smoke: 1 passed, 9 unrelated live cases skipped; real catalog, buffered TTS, WebSocket bytes, and telephony covered.
git diff --check passed.
Fresh full-branch autoreview: 0 actionable findings, confidence 0.95.

Hosted CI was explicitly relaxed for this landing; the PR reports no required checks. Known proof gap: no live Discord upload is claimed. Discord production code is unchanged, and its stream-to-buffer path is covered by 131 focused manager e2e tests.

steipete · 2026-07-13T13:08:32Z

Merged via squash.

Prepared head SHA: 156879a36e6a449b0290d1c41416800e98f9e55a
Landed commit: 93b3f4c45b17ff40d9997e55de4d3a3b13f84bdb

* feat(xai): add streaming TTS via wss://api.x.ai/v1/tts Expose streamSynthesize on the bundled xAI speech provider so Talk and Discord streaming playback can use xAI alongside batch REST synthesis. Adds ws runtime dependency, byte-cap enforcement, and provider tests. * fix(xai): harden streaming TTS per review feedback Restrict streaming WS to api.x.ai, decode WebSocket payloads explicitly, refresh idle timeout on audio.delta, add lifecycle/security tests, and document the official xAI streaming contract plus non-realtime-voice scope. * fix(xai): bound streaming TTS text deltas * fix(xai): require secure native TTS streaming endpoint * docs(tts): remove duplicate output formats section * fix(xai): preserve surrogate pairs in tts frames * fix(xai): bound streaming TTS websocket payloads * fix(xai): align TTS websocket decoding with ws types * fix(xai): pin native streaming TTS endpoint * test(xai): cover native streaming TTS live path * fix(xai): preserve streaming codec compatibility * style(xai): format streaming endpoint guard * docs(tts): describe xAI streaming output policy [skip ci] * fix(xai): own voice-note TTS format policy [skip ci] * docs(tts): keep provider summary concise [skip ci] * test(xai): use explicit placeholder credentials [skip ci] * test(xai): keep fixtures scanner-safe [skip ci] --------- Co-authored-by: TheAngryPit <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>

TheAngryPit requested a review from a team as a code owner July 10, 2026 23:35

openclaw-barnacle Bot added docs Improvements or additions to documentation extensions: tts-local-cli plugin: azure-speech Azure Speech plugin extensions: xai size: L labels Jul 10, 2026

github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 10, 2026

openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 10, 2026

TheAngryPit and others added 11 commits July 11, 2026 08:56

feat(xai): add streaming TTS via wss://api.x.ai/v1/tts

e1f8624

Expose streamSynthesize on the bundled xAI speech provider so Talk and Discord streaming playback can use xAI alongside batch REST synthesis. Adds ws runtime dependency, byte-cap enforcement, and provider tests.

fix(xai): harden streaming TTS per review feedback

fec7164

Restrict streaming WS to api.x.ai, decode WebSocket payloads explicitly, refresh idle timeout on audio.delta, add lifecycle/security tests, and document the official xAI streaming contract plus non-realtime-voice scope.

fix(xai): bound streaming TTS text deltas

2ab87b4

fix(xai): require secure native TTS streaming endpoint

a873d29

docs(tts): remove duplicate output formats section

8901eb9

fix(xai): preserve surrogate pairs in tts frames

23965d5

fix(xai): bound streaming TTS websocket payloads

45cb23e

fix(xai): align TTS websocket decoding with ws types

8be6555

fix(xai): pin native streaming TTS endpoint

755c60d

test(xai): cover native streaming TTS live path

b67baf5

fix(xai): preserve streaming codec compatibility

f08d1fd

TheAngryPit force-pushed the feat/xai-streaming-tts branch from cde93da to f08d1fd Compare July 11, 2026 10:45

openclaw-barnacle Bot added the channel: discord Channel integration: discord label Jul 11, 2026

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 11, 2026

BradGroux mentioned this pull request Jul 11, 2026

WORKING: All Microsoft Issues and PRs (refresh) #74163

Draft

style(xai): format streaming endpoint guard

a1b1595

clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 11, 2026

steipete added 7 commits July 13, 2026 13:27

chore: sync xAI streaming TTS with main [skip ci]

71ad2c9

docs(tts): describe xAI streaming output policy [skip ci]

9f39a47

fix(xai): own voice-note TTS format policy [skip ci]

a8e4e1f

chore: sync xAI streaming TTS with latest main [skip ci]

d4f9b36

docs(tts): keep provider summary concise [skip ci]

0203962

test(xai): use explicit placeholder credentials [skip ci]

a662086

test(xai): keep fixtures scanner-safe [skip ci]

156879a

openclaw-barnacle Bot removed the channel: discord Channel integration: discord label Jul 13, 2026

github-actions Bot removed the dependencies-changed PR changes dependency-related files label Jul 13, 2026

steipete merged commit 93b3f4c into openclaw:main Jul 13, 2026
20 checks passed

steipete mentioned this pull request Jul 13, 2026

feat(xai): add native streaming TTS WebSocket support #99660

Closed

TheAngryPit mentioned this pull request Jul 18, 2026

feat(xai): add native streaming TTS TheAngryPit/openclaw#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(xai): add native streaming TTS#103993

feat(xai): add native streaming TTS#103993
steipete merged 19 commits into
openclaw:mainfrom
TheAngryPit:feat/xai-streaming-tts

TheAngryPit commented Jul 10, 2026 •

edited by steipete

Loading

Uh oh!

github-actions Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

TheAngryPit commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 13, 2026

Uh oh!

Uh oh!

steipete commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

TheAngryPit commented Jul 10, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Scope

Agent Transcript

Uh oh!

github-actions Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency graph guard cleared

Uh oh!

clawsweeper Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheAngryPit commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 13, 2026

Uh oh!

Uh oh!

steipete commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TheAngryPit commented Jul 10, 2026 •

edited by steipete

Loading

github-actions Bot commented Jul 10, 2026 •

edited

Loading

clawsweeper Bot commented Jul 10, 2026 •

edited

Loading

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading