Skip to content

feat(xai): stream TTS over native websocket#99883

Closed
qingminglong wants to merge 4 commits into
openclaw:mainfrom
qingminglong:codex/99660-xai-streaming-tts-resubmit
Closed

feat(xai): stream TTS over native websocket#99883
qingminglong wants to merge 4 commits into
openclaw:mainfrom
qingminglong:codex/99660-xai-streaming-tts-resubmit

Conversation

@qingminglong

@qingminglong qingminglong commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #99660

What Problem This Solves

Fixes an issue where xAI could be used for buffered OpenClaw TTS, but streaming playback callers could not use xAI's native lower-latency streaming TTS WebSocket contract.

Why This Change Was Made

The xAI speech provider now exposes streamSynthesize through the documented native wss://api.x.ai/v1/tts WebSocket flow while leaving existing buffered synthesize and telephony paths on REST /v1/tts. The streaming path uses the Node runtime WebSocket client with xAI authorization headers, and it is restricted to the native xAI host so REST-compatible proxy base URLs are not silently treated as xAI WebSocket endpoints.

User Impact

Streaming TTS callers can receive xAI audio chunks as they arrive instead of waiting for a complete buffered audio response, while existing normal TTS attachment and telephony behavior remains unchanged.

Evidence

  • Claim proved: xAI streaming TTS uses the native WebSocket endpoint, sends ext.delta then ext.done, consumes �udio.delta chunks until �udio.done, and is wired through the speech provider streaming contract.
  • Current branch head SHA: ebe35a4.

  • Commands / artifacts:

ode scripts/run-vitest.mjs extensions/xai/tts.test.ts extensions/xai/speech-provider.test.ts passed at current head: 2 files, 21 tests.

ode scripts/run-vitest.mjs src/plugins/contracts/extension-runtime-dependencies.contract.test.ts passed at current head: 1 file, 404 tests.

ode scripts/run-vitest.mjs test/scripts/plugin-package-dependencies.test.ts passed on the preceding head before the dependency-graph-removal commit: 1 file, 3 tests.

ode scripts/run-tsgo.mjs -p tsconfig.extensions.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/extensions.tsbuildinfo passed at current head.

ode scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.extensions.json extensions/xai/tts.ts passed after the lint follow-up; current-head sgo and dependency-contract proof also cover the final no-ws runtime-import shape.

  • git diff --check upstream/main...HEAD passed at current head.
  • python .agents\skills\autoreview\scripts\autoreview --mode local reported �utoreview clean: no accepted/actionable findings reported for the lint/dependency follow-up patch.
  • Official xAI Text-to-Speech docs checked for the native streaming endpoint and event protocol: https://docs.x.ai/developers/model-capabilities/audio/text-to-speech
  • Observed result: focused tests cover URL/query shape, WebSocket headers, text event ordering, audio chunk decoding, early close, connection error, timeout, release behavior, provider integration, and package dependency ownership. The current head removed the earlier runtime ws package import, so the PR no longer changes extensions/xai/package.json or pnpm-lock.yaml.
  • Not tested / proof gaps: no live xAI credential smoke was run in this environment, so there is no real-service audio proof yet. pnpm docs:list was previously attempted in this worktree family but local pnpm aborted with ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY; the docs diff was checked directly instead. pnpm deps:shrinkwrap:check also aborts in this Codex worktree because
    ode_modules is a junction to the main checkout and pnpm tries to remove it without a TTY; current-head diff no longer includes dependency manifest or lockfile changes, and the focused plugin runtime dependency contract passed.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation extensions: xai size: L labels Jul 4, 2026
@clawsweeper

clawsweeper Bot commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 4, 2026, 5:08 AM ET / 09:08 UTC.

Summary
Adds xAI streamSynthesize support over the native TTS WebSocket endpoint, updates focused xAI speech/TTS tests, and revises the xAI provider docs.

PR surface: Source +344, Tests +289, Docs +3. Total +636 across 5 files.

Reproducibility: not applicable. as a bug reproduction; this is a feature PR. Current-main source inspection confirms xAI lacks streamSynthesize, and the shared streaming path skips providers without that hook.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: extensions/xai/tts.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #99660
Summary: This PR is the active candidate fix for the canonical open xAI native streaming TTS request; the earlier same-author PR was closed unmerged and superseded by this resubmission.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Add redacted live xAI streaming TTS proof showing the WebSocket event flow and non-empty audio bytes; redact API keys, IPs, phone numbers, and non-public endpoints before posting.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body explicitly says no live xAI credential smoke was run; add redacted live output, logs, terminal screenshots, or an artifact showing the real WebSocket event flow and non-empty audio, then update the PR body to trigger re-review.

Risk before merge

  • [P1] No live xAI credential smoke proves the real WebSocket handshake, event flow, and non-empty audio bytes, so unit tests and docs review do not settle whether real xAI streaming TTS works end to end.
  • [P2] Discord voice will treat a successful textToSpeechStream handoff as the playback source; if the live xAI socket only fails after the helper returns, buffered fallback may not run for that reply.

Maintainer options:

  1. Require live xAI streaming proof (recommended)
    Ask for redacted terminal output, logs, or an artifact showing text.delta/text.done leading to audio.delta/audio.done and non-empty audio bytes before merge.
  2. Accept mocked proof only
    Maintainers may intentionally accept the remaining live-service risk if they are comfortable relying on focused unit tests and the upstream docs contract.
  3. Pause until credentials are available
    If no one can safely run a live xAI smoke, keep the PR open rather than merging a streaming path that only mocks the external service.

Next step before merge

  • [P1] The remaining blocker is contributor live-service proof and maintainer merge judgment, not a narrow code defect for an automated repair lane.

Security
Cleared: No concrete security or supply-chain regression was found: the final head has no dependency/lockfile change, reuses existing xAI credential resolution, and restricts the WebSocket path to the native xAI host.

Review details

Best possible solution:

Merge only after redacted live xAI streaming TTS proof shows the native WebSocket path producing non-empty audio while preserving existing buffered TTS behavior.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug reproduction; this is a feature PR. Current-main source inspection confirms xAI lacks streamSynthesize, and the shared streaming path skips providers without that hook.

Is this the best way to solve the issue?

Yes, with a proof gap: the provider-local streamSynthesize implementation is the narrow owner-boundary solution, while the remaining requirement is live xAI streaming proof before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against eafe2a8d0bb6.

Label changes

Label changes:

  • add merge-risk: 🚨 message-delivery: The new streaming path can become the Discord voice playback source before any real xAI audio chunk is proven, so green mocked tests do not fully settle delivered-audio behavior.

Label justifications:

  • P3: This is a useful xAI provider latency feature with limited blast radius; existing buffered xAI TTS and other providers still work.
  • merge-risk: 🚨 message-delivery: The new streaming path can become the Discord voice playback source before any real xAI audio chunk is proven, so green mocked tests do not fully settle delivered-audio behavior.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body explicitly says no live xAI credential smoke was run; add redacted live output, logs, terminal screenshots, or an artifact showing the real WebSocket event flow and non-empty audio, then update the PR body to trigger re-review.
Evidence reviewed

PR surface:

Source +344, Tests +289, Docs +3. Total +636 across 5 files.

View PR surface stats
Area Files Added Removed Net
Source 2 346 2 +344
Tests 2 298 9 +289
Docs 1 27 24 +3
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 5 671 35 +636

What I checked:

  • PR head and discussion: Live PR metadata shows this is an open contributor PR at head ebe35a43dfbaa3a28ad0b0f0d1b22d4791d46fa2, linked to the canonical xAI streaming TTS issue, with the PR body explicitly saying no live xAI credential smoke was run. (ebe35a43dfba)
  • Current main behavior: Current main has the shared streaming TTS hook, and streamSpeech skips providers that do not implement streamSynthesize, which is the gap this PR targets for xAI. (packages/speech-core/src/tts.ts:1636, eafe2a8d0bb6)
  • PR implementation: The PR adds streamSynthesize to the xAI speech provider and forwards the existing xAI provider config, overrides, credential resolution, response format, timeout, stream, and release hook through the new streaming helper. (extensions/xai/speech-provider.ts:256, ebe35a43dfba)
  • WebSocket helper: The PR restricts streaming to native https://api.x.ai/v1, builds wss://api.x.ai/v1/tts query parameters, sends text.delta then text.done, decodes audio.delta, closes on audio.done, and returns a release hook. (extensions/xai/tts.ts:190, ebe35a43dfba)
  • Focused tests: The PR adds fake-WebSocket tests for URL/query shape, Authorization/User-Agent headers, event ordering, audio chunk decoding, non-native baseUrl rejection, early close, timeout, connection errors, and release behavior. (extensions/xai/tts.test.ts:255, ebe35a43dfba)
  • Runtime WebSocket contract probe: Local Node 22.23.1 accepted the native WebSocket(url, { headers }) shape and sent Authorization and User-Agent headers to a localhost handshake probe, matching the PR's no-extra-runtime-dependency approach.

Likely related people:

  • KateWilkins: PR metadata for merged FEAT: Add tts and image generation support to xai extension #68694 lists this account as the author of the xAI media-provider work that added extensions/xai/tts.ts and extensions/xai/speech-provider.ts. (role: introduced xAI buffered TTS surface; confidence: high; commits: f342da5fcc36; files: extensions/xai/tts.ts, extensions/xai/speech-provider.ts, extensions/xai/speech-provider.test.ts)
  • steipete: The commits in merged FEAT: Add tts and image generation support to xai extension #68694 were authored by this account and the PR metadata lists the same account as merger, making them a practical routing candidate for owner-boundary review. (role: commit author and merger for adjacent xAI media-provider work; confidence: medium; commits: 60996d5fb675, 9909151713e6, f342da5fcc36; files: extensions/xai/tts.ts, extensions/xai/speech-provider.ts, docs/providers/xai.md)
  • vincentkoc: Available history shows recent xAI provider documentation and release-tag provenance in the same docs/provider area, including capability-gap documentation that this PR updates. (role: recent xAI provider docs contributor; confidence: medium; commits: 90fac509875e, e085fa1a3ffd; files: docs/providers/xai.md, extensions/xai/speech-provider.ts, extensions/xai/tts.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (4 earlier review cycles)
  • reviewed 2026-07-04T08:13:20.190Z sha 0dbfb2f :: needs real behavior proof before merge. :: [P1] Use a WebSocket client type that accepts headers | [P1] Avoid the impossible readyState comparison
  • reviewed 2026-07-04T08:26:48.889Z sha c61275b :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-04T08:33:19.858Z sha c61275b :: needs real behavior proof before merge. :: [P1] Declare ws as a runtime dependency | [P2] Reject an Error from the connection handler
  • reviewed 2026-07-04T09:01:25.673Z sha ebe35a4 :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jul 4, 2026
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 4, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 4, 2026
@qingminglong
qingminglong requested a review from a team as a code owner July 4, 2026 08:51
@github-actions github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 4, 2026
@github-actions

github-actions Bot commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Dependency graph guard cleared

This PR no longer has blocked dependency graph changes. A future dependency graph change requires a fresh /allow-dependencies-change comment after the guard blocks that new head SHA.

  • Current SHA: ebe35a43dfbaa3a28ad0b0f0d1b22d4791d46fa2

@clawsweeper clawsweeper Bot removed the rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. label Jul 4, 2026
@github-actions github-actions Bot removed the dependencies-changed PR changes dependency-related files label Jul 4, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. and removed merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 4, 2026
This was referenced Jul 6, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docs Improvements or additions to documentation extensions: xai merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: L status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(xai): add native streaming TTS WebSocket support

1 participant

@qingminglong