feat(voiceclaw-realtime): add xAI Realtime Voice Agent provider (closes #73019) by matthewtran172 · Pull Request #73032 · openclaw/openclaw

matthewtran172 · 2026-04-27T20:46:45Z

This is a draft PR opening the design proposed in #73019 for review. Adds support for xAI's Voice Agent API (grok-voice-think-fast-1.0) as a /voiceclaw/realtime provider — the same surface that ships gemini-live.ts today. xAI's Realtime API is documented as OpenAI-Realtime-protocol-compatible (xAI docs); this PR ships the xAI specialization first.

Surface scope

This PR targets /voiceclaw/realtime (gateway endpoint, VoiceClawRealtimeAdapter interface) — the surface for external WebSocket clients. It does not target the separate talk-realtime-relay.ts / src/realtime-voice/ provider-plugin surface used by OpenClaw's browser Talk UI, macOS Talk app, voice-call telephony, and Google Meet integration. Those use a different interface (RealtimeVoiceProviderPlugin) and are wired via api.registerRealtimeVoiceProvider(...) in extension index.ts files (e.g., extensions/openai/realtime-voice-provider.ts).

Related but not addressed by this PR:

feat(talk/macOS): add OpenAI Realtime (speech-to-speech) path for Talk Mode — parity with the voice-call plugin's phone experience #71195 — adds OpenAI Realtime to native macOS Talk via the provider-plugin surface.
Feature: Live Voice Mode in /chat UI (Grok Voice Loop) #12911 — adds Grok voice to /chat web Talk via the provider-plugin surface (would benefit from a parallel extensions/xai/realtime-voice-provider.ts follow-up; happy to scope that separately if maintainers want it).

If maintainers prefer a unified shape across surfaces, I'm open to either (a) extracting a shared OpenAI-Realtime-protocol base used by both xai-realtime.ts here AND a future extensions/xai/realtime-voice-provider.ts, or (b) keeping the two surfaces independent as today.

What this PR does

Adds provider: "xai" as a /voiceclaw/realtime option in the existing VoiceClawSessionConfigEvent.provider union.
Adds src/gateway/voiceclaw-realtime/xai-realtime.ts implementing VoiceClawRealtimeAdapter against the OpenAI-Realtime wire protocol with xAI's documented deltas:
- response.text.delta (xAI) instead of response.output_text.delta.
- Single conversation.item.input_audio_transcription.completed user-transcription event; the adapter synthesizes OpenClaw's transcript.delta + transcript.done pair.
Extends session.ts with provider-aware dispatch:
- resolveProvider(config.provider) — returns "xai" or "gemini" (back-compat default).
- defaultVoiceFor(provider) — "ara" for xAI, "Zephyr" for Gemini.
- requiredApiKeyEnvFor(provider) — XAI_API_KEY or GEMINI_API_KEY.
- createDefaultAdapterFactory() — instantiates VoiceClawXaiRealtimeAdapter or VoiceClawGeminiLiveAdapter based on the resolved provider.
Default voice for xAI is ara; all five xAI voices (eve, ara, rex, sal, leo) are supported with case-insensitive resolution and per-session override.
Updates docs/gateway/index.md (provider option) and docs/providers/xai.md (realtime row in the feature-coverage table; cross-link from "Known limits").

What this PR does not do

Does not modify gemini-live.ts (Gemini behavior preserved byte-for-byte).
Does not modify extensions/voice-call/, extensions/openai/realtime-voice-provider.ts, or extensions/xai/ batch/streaming-STT files (separate surfaces).
Does not change gateway auth, sandboxing, or the WS control protocol.
Does not add any new HTTP endpoint.
Does not add OpenAI Realtime support to /voiceclaw/realtime (the type union already accepts "openai"; the adapter is intentionally a separate, smaller PR).
Does not add an xAI RealtimeVoiceProviderPlugin for the provider-plugin surface (surface B). Happy to follow up if maintainers want it.

Security

XAI_API_KEY is read from the gateway process environment at upstream-open time; never logged, never reflected in client errors, never persisted on the adapter instance.
Sanitization defends against Bearer ... token reflection and xai-... key reflection in upstream error messages.
Test asserts XAI_API_KEY value never appears in log lines or in JSON.stringify(adapter).

Tests

Mocked WebSocket only; no live xAI calls in CI; no XAI_API_KEY value required to run tests (a fake TEST_KEY_NOT_REAL is used in tests where adapter construction is exercised).
Coverage:
- Voice helpers: default = ara, all five voices accepted case-insensitively, unknown voice falls back to ara, isValidXaiVoice correctness.
- Audio passthrough: response.output_audio.delta and OpenAI-style response.audio.delta both forward as OpenClaw audio.delta.
- Transcript: xAI response.text.delta → OpenClaw assistant transcript.delta. Single user-transcription completed event → synthesized turn.started + transcript.delta + transcript.done.
- Barge-in: input_audio_buffer.speech_started mid-assistant-response → finalizes pending assistant text with ellipsis + emits turn.started (user role).
- Function calls: response.function_call_arguments.done → tool.call. Tool result returned via conversation.item.create (with function_call_output) + response.create.
- Audio frame forwarding: audio.append → input_audio_buffer.append.
- Turn end: response.done → flush transcripts + turn.ended.
- Usage metrics: response.usage translated to OpenClaw usage.metrics.
- Error sanitization: xAI error reflecting Bearer xai-supersecretvalue123 → message contains Bearer ***, never the raw value.
- Unknown event types ignored without throwing.
- session.update payload includes resolved voice (ara by default), modalities ["text", "audio"], pcm16 audio formats, server_vad turn detection.
- Function tools serialized into session.update when registered.
- Adapter constructible without XAI_API_KEY set (key is only required at openUpstream).
- Adapter factory dispatches by provider: "xai" → VoiceClawXaiRealtimeAdapter; "gemini" and undefined → VoiceClawGeminiLiveAdapter.

Local gates run on this branch

pnpm exec vitest run src/gateway/voiceclaw-realtime/   →  120 tests passed (15 files)
pnpm exec oxlint src/gateway/voiceclaw-realtime/       →  0 warnings, 0 errors
pnpm tsgo:core                                          →  clean
pnpm tsgo:core:test                                     →  clean

No live calls, no openclaw onboard, no installs/upgrades beyond pnpm install --frozen-lockfile.

SAGE is not changed

This PR is OpenClaw upstream only. The downstream SAGE project mentioned in #73019 will integrate against this provider once it ships; no SAGE code is included here.

Design notes for review

I'd appreciate maintainer input on the Questions for maintainers section in #73019, particularly:

Single-shared-base vs separate-adapters preference.
Ship order — happy to bundle OpenAI Realtime on /voiceclaw/realtime in this PR if that's preferred, or keep PR small as proposed.
Plugin-SDK extension vs core gateway code (the gemini-live.ts precedent suggests core; happy to move).
Whether to also follow up with an extensions/xai/realtime-voice-provider.ts for the surface-B plugin path (would close Feature: Live Voice Mode in /chat UI (Grok Voice Loop) #12911).

The adapter avoids modifying gemini-live.ts to keep the Gemini path byte-stable. If maintainers prefer a shared base in this PR, I can refactor.

🤖 Generated with Claude Code

Implements Issue openclaw#73019. Adds xAI's Voice Agent API (grok-voice-think-fast-1.0) as a /voiceclaw/realtime provider. xAI's Realtime API is documented as OpenAI-Realtime-protocol-compatible [1]; this adapter handles the two documented wire deltas: 1. response.text.delta (xAI) vs response.output_text.delta (OpenAI) 2. Single conversation.item.input_audio_transcription.completed event for user transcription (vs OpenAI's split delta/completed pair). The adapter synthesizes OpenClaw's transcript.delta + transcript.done pair from the single completed event. OpenAI Realtime support is anticipated by the existing types.ts provider union but is intentionally out of scope for this PR — see issue openclaw#71195. The adapter is structured so an OpenAI Realtime specialization could share most wire-protocol logic in a follow-up. Provider selection is via session.config.provider="xai". Default voice is "ara"; all five xAI voices (eve, ara, rex, sal, leo) accepted with case-insensitive resolution. Per-session voice override supported. XAI_API_KEY is read from the gateway process environment only; never logged, never reflected in client-facing errors. Sanitization defends against Bearer token / xai-* key reflection in upstream errors. Files added: - src/gateway/voiceclaw-realtime/xai-realtime.ts - src/gateway/voiceclaw-realtime/xai-realtime.test.ts Files modified: - src/gateway/voiceclaw-realtime/types.ts (add "xai" to provider union) - src/gateway/voiceclaw-realtime/session.ts (provider dispatch + per-provider voice/key defaults; back-compat default = "gemini") - src/gateway/voiceclaw-realtime/session.test.ts (provider selection) - docs/gateway/index.md (document xai provider option) - docs/providers/xai.md (note realtime support) Tests: mocked WebSocket only, no live xAI in CI, no API key required. Coverage: voice resolution, audio passthrough, transcript synthesis, function-call flow, barge-in, error sanitization, secret discipline, provider selection, default voice ara. Local gates run on this branch: - pnpm exec vitest run src/gateway/voiceclaw-realtime/ (120 passed) - pnpm exec oxlint src/gateway/voiceclaw-realtime/ (0 errors) - pnpm tsgo:core (clean) - pnpm tsgo:core:test (clean) [1] https://docs.x.ai/developers/model-capabilities/audio/voice-agent Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Self-audit follow-up for openclaw#73019 / PR openclaw#73032. Adds tests for previously uncovered adapter behaviors: Client-side methods: - commitAudio forwards input_audio_buffer.commit - createResponse forwards response.create - cancelResponse forwards response.cancel best-effort - injectContext forwards system-role conversation.item.create - sendFrame drops video frames silently (xAI Voice Agent is audio-only) - getTranscript returns a copy (caller mutation isolated) - disconnect clears state and is idempotent Close-handling: - code 1000 ignored without surfacing error - non-reconnectable code emits sanitized error - reconnectable code triggers reconnect path - in-flight tool calls are cancelled with explicit error if upstream closes mid-tool-call Conversation history replay: - last 12 entries serialized via conversation.item.create - user role serializes content type as input_text; assistant as text - empty / missing conversationHistory is a no-op No code changes in this commit; tests only. No live calls; no XAI_API_KEY required (mocks bypass openUpstream). Local gates: - pnpm exec vitest run src/gateway/voiceclaw-realtime/ -> 159 tests passed (was 120) - pnpm exec oxlint src/gateway/voiceclaw-realtime/ -> 0 errors - pnpm tsgo:core -> clean - pnpm tsgo:core:test -> clean Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

clawsweeper · 2026-04-30T15:37:58Z

Thanks for the contribution. I reviewed the branch, and this PR is not a good landing base for OpenClaw.

Close this PR as no longer a useful landing branch: the xAI realtime voice idea is useful, but current main has moved realtime voice onto the talk.realtime provider-plugin surface, while this branch targets the removed src/gateway/voiceclaw-realtime stack, is GitHub-conflicting, lacks real xAI proof, and mixes unrelated iOS/maintenance work.

So I’m closing this PR rather than keeping an unmergeable branch open. A new narrow PR that carries only the useful part is welcome.

Review details

Best possible solution:

Restart the useful xAI realtime voice work as a narrow extensions/xai RealtimeVoiceProviderPlugin/talk.realtime PR, with unrelated iOS and maintenance edits split out.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a feature PR. Source checks show the branch no longer matches current main: the targeted gateway module is gone and current xAI docs still list realtime voice as unregistered.

Is this the best way to solve the issue?

No. The xAI realtime goal is useful, but this branch targets a removed /voiceclaw/realtime stack; the maintainable fix is a narrow xAI realtime voice provider for the current talk.realtime contracts.

Security review:

Security review cleared: No concrete security or supply-chain regression was found in the diff, but the provider credential path still needs real runtime proof before any replacement PR lands.

AGENTS.md: found and applied where relevant.

What I checked:

current main realtime voice entry point: Current main creates realtime Talk relay sessions from a RealtimeVoiceProviderPlugin, not from the deleted voiceclaw-realtime gateway adapter stack. (src/gateway/talk-realtime-relay.ts:397, 66880a5d736c)
current xAI docs still say realtime voice is not exposed: Current provider docs explicitly list xAI Realtime voice as not exposed and say it is not registered as an OpenClaw provider yet. Public docs: docs/providers/xai.md. (docs/providers/xai.md:165, 66880a5d736c)
current xAI plugin contract lacks realtime voice: The bundled xAI plugin manifest advertises model/search/media/speech/realtime-transcription/image/tool contracts, but no realtime voice provider contract. (extensions/xai/openclaw.plugin.json:144, 66880a5d736c)
PR retargeting mismatch: The PR hard-wires VoiceClawXaiRealtimeAdapter into src/gateway/voiceclaw-realtime/session.ts, a path that does not exist on current main. (src/gateway/voiceclaw-realtime/session.ts:379, 8c60fb4013aa)
live PR state: GitHub reports the PR as CONFLICTING/DIRTY; its checks include repeated failing Real behavior proof runs, and the body says only mocked WebSocket tests were run with no live xAI calls. (8c60fb4013aa)
related maintainer direction: Related realtime Talk discussion says the backend stack is mostly present and the merge target should reuse existing realtime Talk session flow rather than add another parallel realtime provider architecture.

Likely related people:

steipete: Authored recent realtime provider-boundary refactors and left related realtime Talk guidance to reuse the existing Talk realtime session flow instead of adding a parallel gateway voice stack. (role: recent realtime voice boundary contributor and scope commenter; confidence: high; commits: a23ab9b906dc, 7cd40ad5654a, a5b563280910; files: extensions/openai/realtime-voice-provider.ts, src/gateway/talk-realtime-relay.ts, src/talk/provider-resolver.ts)
Vincent Koc: Recent provider-docs work is part of the history for the current xAI capability coverage that still marks realtime voice as not exposed. (role: provider docs contributor; confidence: medium; commits: 90fac509875e; files: docs/providers/xai.md)

Codex review notes: model internal, reasoning high; reviewed against 66880a5d736c.

Lets the operator share audio files (voice memos, music clips) via the iOS Share sheet into OpenClaw alongside the existing image / URL / text paths. Mirrors the existing image-attachment plumbing: - extractedContent now handles UTType.audio providers (max 1 audio per share, 5 MB cap) - loadAudioAttachment + preferredAudioTypeIdentifier helpers - preferredFileName + ensureFileExtension helpers (suggestedName fallback to "shared-audio-N.m4a") - composeDraft drops a "Shared audio: <filename>" line when text/title/url all empty - log line now reads "attachments=N" (was image-only count) Audio attachment is base64-encoded into ShareAttachment.content, matching image flow.

Acted on the SAGE-side optimization audit (~/Documents/sage/self-improvement-audit/report.md, openclaw section). openclaw#1 — xai-realtime.ts:719 — replace per-overflow O(n) `.shift()` with batched 25% splice. Hot voice path; under sustained overflow the old shift-per-push was N*overflow_count. Batch trim drops 25% in one splice so amortized per-push cost is O(1) (audio overflow always meant data loss anyway; trimming a quarter at once is cleaner than dropping one frame at a time). openclaw#7 — server-maintenance.ts:93 — `[...map.entries()].toSorted(...)` allocates two distinct copies (spread + toSorted internal). Switched to single Array.from(...) + in-place .sort(). Drops one allocation per maintenance tick. openclaw#8 — xai-realtime.ts:698 + isValidXaiVoice — `.find()` / `.includes()` over the const XAI_VOICES tuple was O(N) per voice resolution. Pre-built `_XAI_VOICE_SET = new Set(XAI_VOICES)`; lookups are now O(1). Skipped openclaw#2 (audit suggested merging 4 server-maintenance Map iterations into one — they touch DIFFERENT collections; merging would be artificial code, not real perf). Multi-day refactors of attempt.ts / loader.ts / chat.ts deferred per audit recommendation.

openclaw-barnacle · 2026-05-30T04:41:48Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

openclaw-barnacle · 2026-06-14T05:03:07Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

clawsweeper · 2026-06-14T05:16:09Z

ClawSweeper applied the proposed close for this PR.

Action: closed this PR.
Close reason: low-signal unmergeable PR.
Evidence: durable ClawSweeper review.

openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime size: XL labels Apr 27, 2026

matthewtran172 marked this pull request as ready for review May 3, 2026 08:28

openclaw-barnacle Bot added the app: ios App: ios label May 3, 2026

openclaw-barnacle Bot added stale Marked as stale due to inactivity triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 30, 2026

clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label May 30, 2026

barnacle-openclaw Bot removed the stale Marked as stale due to inactivity label May 30, 2026

openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 14, 2026

clawsweeper Bot closed this Jun 14, 2026

clawsweeper Bot mentioned this pull request Jul 1, 2026

feat(xai): add realtime voice provider #91308

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(voiceclaw-realtime): add xAI Realtime Voice Agent provider (closes #73019)#73032

feat(voiceclaw-realtime): add xAI Realtime Voice Agent provider (closes #73019)#73032
matthewtran172 wants to merge 4 commits into
openclaw:mainfrom
matthewtran172:phase-22a-xai-realtime-voiceclaw

matthewtran172 commented Apr 27, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Apr 30, 2026 •

edited

Loading

Uh oh!

openclaw-barnacle Bot commented May 30, 2026

Uh oh!

openclaw-barnacle Bot commented Jun 14, 2026

Uh oh!

clawsweeper Bot commented Jun 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

matthewtran172 commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Surface scope

What this PR does

What this PR does not do

Security

Tests

Local gates run on this branch

SAGE is not changed

Design notes for review

Uh oh!

clawsweeper Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openclaw-barnacle Bot commented May 30, 2026

Uh oh!

openclaw-barnacle Bot commented Jun 14, 2026

Uh oh!

clawsweeper Bot commented Jun 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

matthewtran172 commented Apr 27, 2026 •

edited

Loading

clawsweeper Bot commented Apr 30, 2026 •

edited

Loading