Skip to content

feat(xai): add native streaming TTS WebSocket support #99660

Description

@TheAngryPit

Summary

OpenClaw's bundled xAI provider supports buffered TTS through the REST /v1/tts path, but the provider does not currently expose xAI's native streaming TTS WebSocket contract to OpenClaw streaming playback callers.

This leaves streaming-capable surfaces, such as Discord voice paths that use textToSpeechStream, unable to use xAI's lower-latency streamed audio path even when xAI credentials and normal buffered xAI TTS are configured.

Desired behavior

Add native xAI streaming TTS support for OpenClaw's shared speech provider streaming contract:

  • Use xAI's documented WebSocket streaming TTS endpoint: wss://api.x.ai/v1/tts.
  • Send text.delta followed by text.done.
  • Read audio.delta chunks until audio.done.
  • Preserve the existing buffered synthesize path for normal TTS attachments and talk.speak.
  • Keep this separate from xAI realtime voice / Talk gateway-relay, which uses wss://api.x.ai/v1/realtime and has a different session contract.

Scope

In scope:

  • messages.tts.provider: "xai" streaming support for textToSpeechStream callers.
  • Existing OpenClaw TTS controls: language, voice, codec/response format, and speed.
  • Native xAI WebSocket endpoint only, to avoid silently applying the xAI streaming contract to arbitrary REST-compatible base URLs.
  • Focused unit tests for URL shape, event protocol, lifecycle/release behavior, timeout behavior, and streamSynthesize integration.

Out of scope:

  • xAI realtime voice / Talk gateway-relay provider.
  • STT changes.
  • Voice Call realtime/managed-room behavior.
  • New sample_rate / bit_rate config knobs. xAI supports them, but this first pass uses provider defaults rather than extending the OpenClaw config surface.

Why this matters

For real voice use, buffered TTS adds avoidable latency because playback waits for a complete synthesized buffer. xAI's streaming TTS API can start delivering audio chunks earlier, which is important for lower-cost and lower-latency voice setups.

This also lets OpenClaw use xAI for streaming playback without overloading the separate realtime voice feature scope.

Proof target

Minimum acceptable proof for a PR:

  • Focused tests for extensions/xai/tts.ts and extensions/xai/speech-provider.ts.
  • Docs clearly distinguishing streaming TTS from realtime voice.
  • Security posture explaining why streaming WebSocket is restricted to the native xAI host.
  • If live credentials are available, a redacted smoke showing text.delta / text.done -> audio.delta / audio.done and non-empty audio bytes.

Related context

  • Existing xAI TTS provider support exists for buffered REST TTS.
  • xAI realtime voice / Talk gateway-relay is tracked separately and should not be conflated with this feature.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low-priority cleanup, docs, polish, ergonomics, or speculative work.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions