Skip to content

Commit bca345b

Browse files
committed
fix(xai): narrow realtime voice provider scope
1 parent e29f2a3 commit bca345b

31 files changed

Lines changed: 119 additions & 2740 deletions
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
d9dfe6587fcf5a61a61c690e1554996acea9ca2de6d8ac83dfde8c6a4ece18fe plugin-sdk-api-baseline.json
2-
acc91d22c0f35efef663e39c9b6659b71f226a1859455a071f55f641593ff1db plugin-sdk-api-baseline.jsonl
1+
1cd5bcc75461c64d39a00918a50d033e66ae7ec199d8029f7cccaaa2eeb16f22 plugin-sdk-api-baseline.json
2+
a5d3b43c3710c4238958b1b3163e652ac34bdc7b82215c6294ce61b72188d75e plugin-sdk-api-baseline.jsonl

docs/concepts/model-providers.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -337,7 +337,7 @@ See [/providers/kilocode](/providers/kilocode) for setup details.
337337
Model ids use a `nvidia/<vendor>/<model>` namespace (for example `nvidia/nvidia/nemotron-...` alongside `nvidia/moonshotai/kimi-k2.5`); pickers preserve the literal `<provider>/<model-id>` composition while the canonical key sent to the API stays single-prefixed.
338338
</Accordion>
339339
<Accordion title="xAI">
340-
Uses the xAI Responses path. The recommended path is SuperGrok/X Premium OAuth; API keys still work via `XAI_API_KEY` or plugin config, and Grok `web_search` reuses the same auth profile before API-key fallback. `grok-4.3` is the bundled public API default chat model, while `grok-composer-2.5-fast` and `grok-build-0.1` are selectable for Grok OAuth / Grok Build coding workloads. `/fast` or `params.fastMode: true` rewrites `grok-3`, `grok-3-mini`, `grok-4`, and `grok-4-0709` to their `*-fast` variants. `tool_stream` defaults on; disable via `agents.defaults.models["xai/<model>"].params.tool_stream=false`.
340+
Uses the xAI Responses path. The recommended path is SuperGrok/X Premium OAuth; API keys still work via `XAI_API_KEY` or plugin config, and Grok `web_search` reuses the same auth profile before API-key fallback. `grok-4.3` is the bundled default chat model, and `grok-build-0.1` is selectable for build/coding-focused work. `/fast` or `params.fastMode: true` rewrites `grok-3`, `grok-3-mini`, `grok-4`, and `grok-4-0709` to their `*-fast` variants. `tool_stream` defaults on; disable via `agents.defaults.models["xai/<model>"].params.tool_stream=false`.
341341
</Accordion>
342342
<Accordion title="Cerebras">
343343
Ships as the bundled `cerebras` provider plugin. GLM uses `zai-glm-4.7`; OpenAI-compatible base URL is `https://api.cerebras.ai/v1`.

docs/nodes/talk.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -116,13 +116,13 @@ Defaults:
116116
- `providers.elevenlabs.apiKey`: falls back to `ELEVENLABS_API_KEY` (or gateway shell profile if available).
117117
- `consultThinkingLevel`: optional thinking level override for the full OpenClaw agent run behind realtime `openclaw_agent_consult` calls.
118118
- `consultFastMode`: optional fast-mode override for realtime `openclaw_agent_consult` calls.
119-
- `realtime.provider`: selects the active browser/server realtime voice provider. Use `openai` for WebRTC, `google` for provider WebSocket, `xai` for Grok Voice through provider WebSocket or Gateway relay, or another bridge-only provider through Gateway relay.
119+
- `realtime.provider`: selects the active browser/server realtime voice provider. Use `openai` for WebRTC, `google` for provider WebSocket, `xai` for Grok Voice through Gateway relay, or another bridge-only provider through Gateway relay.
120120
- `realtime.providers.<provider>` stores provider-owned realtime config. The browser receives only ephemeral or constrained session credentials, never a standard API key.
121121
- `realtime.providers.openai.voice`: built-in OpenAI Realtime voice id. Current `gpt-realtime-2` voices are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin`, and `cedar`; `marin` and `cedar` are recommended for best quality.
122122
- `realtime.providers.xai.voice` / `speakerVoice` / `voiceId` / `speakerVoiceId`: xAI Grok Voice id. Built-in xAI voices include `eve`, `ara`, `rex`, `sal`, and `leo`; default is `leo`.
123123
- `realtime.transport`: `webrtc` and `provider-websocket` are browser realtime transports. Android uses realtime relay only when this is `gateway-relay`; otherwise Android Talk uses its native STT/TTS loop.
124-
- `realtime.providers.xai.vadThreshold`, `silenceDurationMs`, and `prefixPaddingMs`: forwarded to xAI server VAD for provider WebSocket and Gateway relay sessions.
125-
- `realtime.providers.xai.baseUrl`: custom xAI-compatible Realtime base URL for `gateway-relay` only. Browser-owned xAI `provider-websocket` sessions require the native `https://api.x.ai/v1` endpoint.
124+
- `realtime.providers.xai.vadThreshold`, `silenceDurationMs`, and `prefixPaddingMs`: forwarded to xAI server VAD for Gateway relay sessions.
125+
- `realtime.providers.xai.baseUrl`: custom xAI-compatible Realtime base URL for `gateway-relay`.
126126
- `realtime.brain`: `agent-consult` routes realtime tool calls through Gateway policy; `direct-tools` is legacy direct-tool compatibility behavior; `none` is for transcription or external orchestration.
127127
- `realtime.consultRouting`: `provider-direct` preserves the provider's direct reply when it skips `openclaw_agent_consult`; `force-agent-consult` makes Gateway relay route finalized user transcripts through OpenClaw instead.
128128
- `realtime.instructions`: appends provider-facing system instructions to OpenClaw's built-in realtime prompt. Use it for voice style and tone; OpenClaw keeps the default `openclaw_agent_consult` guidance.

docs/plugins/sdk-provider-plugins.md

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -791,8 +791,6 @@ API key auth, and dynamic model resolution.
791791
connect: async () => {},
792792
sendAudio: () => {},
793793
setMediaTimestamp: () => {},
794-
sendUserMessage: () => {},
795-
speakText: () => {},
796794
handleBargeIn: () => {},
797795
submitToolResult: () => {},
798796
acknowledgeMark: () => {},
@@ -806,13 +804,7 @@ API key auth, and dynamic model resolution.
806804
transports, audio formats, and feature flags to browser and native Talk
807805
clients. Implement `handleBargeIn` when a transport can detect that a
808806
human is interrupting assistant playback and the provider supports
809-
truncating or clearing the active audio response. Implement `speakText`
810-
when the provider can schedule assistant speech from OpenClaw-owned text;
811-
`RealtimeVoiceSpeakTextOptions.mode: "exact"` means the provider should
812-
preserve wording, while `"natural"` allows provider-side verbal polish.
813-
Providers that omit `speakText` should implement `sendUserMessage` so
814-
Gateway relay sessions can fall back to normal provider response
815-
generation.
807+
truncating or clearing the active audio response.
816808
</Tab>
817809
<Tab title="Media understanding">
818810
```typescript

docs/providers/xai.md

Lines changed: 16 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -127,22 +127,20 @@ remote process polls xAI for the completed token exchange.
127127
OpenClaw includes the current xAI chat models out of the box, ordered newest
128128
first in model pickers:
129129

130-
| Family | Model ids |
131-
| -------------------- | ------------------------------------------------------------------------ |
132-
| Composer 2.5 (OAuth) | `grok-composer-2.5-fast` |
133-
| Grok Build 0.1 | `grok-build-0.1` |
134-
| Grok 4.3 | `grok-4.3` |
135-
| Grok 4.20 Beta | `grok-4.20-beta-latest-reasoning`, `grok-4.20-beta-latest-non-reasoning` |
130+
| Family | Model ids |
131+
| -------------- | ------------------------------------------------------------------------ |
132+
| Grok Build 0.1 | `grok-build-0.1` |
133+
| Grok 4.3 | `grok-4.3` |
134+
| Grok 4.20 Beta | `grok-4.20-beta-latest-reasoning`, `grok-4.20-beta-latest-non-reasoning` |
136135

137136
The plugin still forward-resolves older Grok 3, Grok 4, Grok 4 Fast, Grok 4.1
138137
Fast, and Grok Code slugs for existing configs. Official Grok Code Fast aliases
139138
normalize to `grok-build-0.1`; OpenClaw no longer shows the other retired
140139
upstream slugs in the selectable catalog.
141140

142141
<Tip>
143-
Use `grok-4.3` for the public xAI API default. Use `grok-composer-2.5-fast`
144-
or `grok-build-0.1` for Grok OAuth / Grok Build coding workloads unless you
145-
explicitly need a Grok 4.20 beta alias.
142+
Use `grok-4.3` for general chat and `grok-build-0.1` for build/coding-focused
143+
workloads unless you explicitly need a Grok 4.20 beta alias.
146144
</Tip>
147145

148146
## OpenClaw feature coverage
@@ -424,22 +422,16 @@ Legacy aliases still normalize to the canonical bundled ids:
424422

425423
<Accordion title="Realtime voice">
426424
The bundled `xai` plugin registers a realtime voice provider for Talk
427-
browser WebSocket and Gateway relay sessions.
425+
Gateway relay sessions.
428426

429427
- Provider id: `xai`
430-
- Transports: `provider-websocket`, `gateway-relay`
428+
- Transports: `gateway-relay`
431429
- Default model: `grok-voice-latest`
432430
- Models: `grok-voice-latest`, `grok-voice-think-fast-1.0`,
433431
`grok-voice-fast-1.0`
434432
- Default voice: `leo`
435433
- Voices: `eve`, `ara`, `rex`, `sal`, `leo`, or another xAI voice id
436434
- Audio formats: PCM16 24 kHz or G.711 µ-law 8 kHz
437-
- Browser auth: OpenClaw creates an ephemeral xAI Realtime client secret
438-
with `POST /v1/realtime/client_secrets`; the browser connects with the
439-
`xai-client-secret.<token>` WebSocket subprotocol.
440-
- Browser transport origin: `provider-websocket` is limited to the native
441-
`https://api.x.ai/v1` Realtime endpoint. Use `gateway-relay` for custom
442-
`baseUrl` or proxy endpoints.
443435
- Inbound speech transcripts: xAI's cumulative
444436
`conversation.item.input_audio_transcription.updated` event is exposed as
445437
user transcript updates, so Talk and meeting surfaces can consume live
@@ -453,7 +445,7 @@ Legacy aliases still normalize to the canonical bundled ids:
453445
model: "grok-voice-latest",
454446
speakerVoice: "leo",
455447
mode: "realtime",
456-
transport: "provider-websocket",
448+
transport: "gateway-relay",
457449
brain: "agent-consult",
458450
providers: {
459451
xai: {
@@ -473,15 +465,12 @@ Legacy aliases still normalize to the canonical bundled ids:
473465
`voiceId`, `speakerVoiceId`, `vadThreshold`, `silenceDurationMs`,
474466
`prefixPaddingMs`, `interruptResponseOnInputAudio`, and
475467
`minBargeInAudioEndMs`. When `apiKey` is omitted, OpenClaw tries
476-
`XAI_API_KEY` and then the configured xAI auth profile. `baseUrl` is a
477-
server-side option for `gateway-relay`; browser-owned `provider-websocket`
478-
sessions require the native xAI Realtime endpoint.
468+
`XAI_API_KEY` and then the configured xAI auth profile.
479469

480470
<Note>
481-
Browser-owned xAI Realtime sessions use ephemeral client secrets. Gateway
482-
relay remains available when you need the vendor WebSocket to stay
483-
server-side, such as native clients, backend-owned meeting capture, or
484-
custom/proxy `baseUrl` deployments.
471+
Gateway relay keeps the vendor WebSocket server-side, which is the supported
472+
xAI realtime voice path for native clients, backend-owned meeting capture,
473+
and custom/proxy `baseUrl` deployments.
485474
</Note>
486475

487476
</Accordion>
@@ -570,10 +559,8 @@ Legacy aliases still normalize to the canonical bundled ids:
570559
- `grok-4.20-multi-agent-experimental-beta-0304` is not supported on the
571560
normal xAI provider path because it requires a different upstream API
572561
surface than the standard OpenClaw xAI transport.
573-
- xAI Realtime voice supports Talk Gateway relay and browser-owned
574-
provider-WebSocket sessions. Custom or proxy `baseUrl` values are
575-
Gateway relay only; browser-owned provider-WebSocket sessions require
576-
xAI's native `https://api.x.ai/v1` endpoint.
562+
- xAI Realtime voice supports Talk Gateway relay. Browser-owned
563+
provider-WebSocket sessions are not registered for xAI.
577564
- xAI image `quality`, image `mask`, and extra native-only aspect ratios are
578565
not exposed until the shared `image_generate` tool has corresponding
579566
cross-provider controls.

docs/web/control-ui.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -201,12 +201,10 @@ Activity entries keep only sanitized summaries and redacted, truncated output pr
201201

202202
</Accordion>
203203
<Accordion title="Talk mode (browser realtime)">
204-
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key auth profile, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`; OpenAI OAuth profiles do not configure Realtime voice. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. Configure xAI Grok Voice with `talk.realtime.provider: "xai"` plus either `talk.realtime.providers.xai.apiKey`, `XAI_API_KEY`, or an xAI auth profile. The browser never receives a standard provider API key. OpenAI receives an ephemeral Realtime client secret for WebRTC. Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. xAI receives an ephemeral Realtime client secret from `/v1/realtime/client_secrets` and connects to the Grok Voice WebSocket with the `xai-client-secret.<token>` subprotocol. Browser-owned xAI sessions require the native `https://api.x.ai/v1` endpoint; use Gateway relay for custom xAI `baseUrl` or proxy endpoints. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
204+
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key auth profile, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`; OpenAI OAuth profiles do not configure Realtime voice. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. The browser never receives a standard provider API key. OpenAI receives an ephemeral Realtime client secret for WebRTC. Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
205205

206206
The Chat composer includes a Talk options button next to the Talk start/stop button. The options apply to the next Talk session and can override provider, transport, model, voice, reasoning effort, VAD threshold, silence duration, and prefix padding. When an option is blank, the Gateway uses configured defaults where available or the provider default. Selecting Gateway relay forces the backend relay path; selecting WebRTC keeps the session client-owned and fails instead of silently falling back to relay if the provider cannot create a browser session.
207207

208-
Provider WebSocket and Gateway relay providers can emit inbound transcript updates while audio is still streaming. For xAI, the bridge maps the cumulative `conversation.item.input_audio_transcription.updated` event to user transcript updates so Talk and meeting transcription surfaces can consume live inbound speech content.
209-
210208
In the Chat composer, the Talk control is the waves button next to the microphone dictation button. When Talk starts, the composer status row shows `Connecting Talk...`, then `Talk live` while audio is connected, or `Asking OpenClaw...` while a realtime tool call is consulting the configured larger model through `talk.client.toolCall`.
211209

212210
Maintainer live smoke: `OPENAI_API_KEY=... GEMINI_API_KEY=... node --import tsx scripts/dev/realtime-talk-live-smoke.ts` verifies the OpenAI backend WebSocket bridge, OpenAI browser WebRTC SDP exchange, Google Live constrained-token browser WebSocket setup, and the Gateway relay browser adapter with fake microphone media. The command prints provider status only and does not log secrets.

extensions/xai/index.test.ts

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -289,7 +289,6 @@ describe("xai provider plugin", () => {
289289
expect(result.provider.auth).toBe("oauth");
290290
expect(result.provider.apiKey).toBeUndefined();
291291
expect(result.provider.models.map((model) => model.id)).toContain("grok-build-0.1");
292-
expect(result.provider.models.map((model) => model.id)).toContain("grok-composer-2.5-fast");
293292
});
294293

295294
it("falls back to API-key discovery when xAI OAuth credential resolution fails", async () => {

extensions/xai/model-definitions.ts

Lines changed: 1 addition & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -3,18 +3,15 @@ import type { ModelDefinitionConfig } from "openclaw/plugin-sdk/provider-model-s
33
import { normalizeOptionalLowercaseString } from "openclaw/plugin-sdk/string-coerce-runtime";
44

55
export const XAI_BASE_URL = "https://api.x.ai/v1";
6-
export const XAI_GROK_OAUTH_BASE_URL = "https://cli-chat-proxy.grok.com/v1";
76
export const XAI_DEFAULT_IMAGE_MODEL = "grok-imagine-image";
87
export const XAI_IMAGE_MODELS = ["grok-imagine-image", "grok-imagine-image-quality"] as const;
98
export const XAI_DEFAULT_CONTEXT_WINDOW = 1_000_000;
109
const XAI_LARGE_CONTEXT_WINDOW = 2_000_000;
1110
const XAI_GROK_4_CONTEXT_WINDOW = 256_000;
1211
const XAI_CODE_CONTEXT_WINDOW = 256_000;
13-
const XAI_COMPOSER_CONTEXT_WINDOW = 200_000;
1412
export const XAI_DEFAULT_MAX_TOKENS = 64_000;
1513
const XAI_LEGACY_CONTEXT_WINDOW = 131_072;
1614
const XAI_LEGACY_MAX_TOKENS = 8_192;
17-
const XAI_COMPOSER_MAX_TOKENS = 30_000;
1815
export const XAI_DEFAULT_MODEL_ID = "grok-4.3";
1916
export const XAI_DEFAULT_MODEL_REF = `xai/${XAI_DEFAULT_MODEL_ID}`;
2017

@@ -23,8 +20,6 @@ type XaiCost = ModelDefinitionConfig["cost"];
2320
type XaiCatalogEntry = {
2421
id: string;
2522
name: string;
26-
api?: ModelDefinitionConfig["api"];
27-
baseUrl?: string;
2823
reasoning: boolean;
2924
input?: ModelDefinitionConfig["input"];
3025
contextWindow: number;
@@ -74,27 +69,6 @@ const XAI_CODE_FAST_COST = {
7469
cacheWrite: 0,
7570
} satisfies XaiCost;
7671

77-
const XAI_SUBSCRIPTION_COST_UNKNOWN = {
78-
input: 0,
79-
output: 0,
80-
cacheRead: 0,
81-
cacheWrite: 0,
82-
} satisfies XaiCost;
83-
84-
const XAI_OAUTH_MODEL_CATALOG = [
85-
{
86-
id: "grok-composer-2.5-fast",
87-
name: "Composer 2.5",
88-
api: "openai-responses",
89-
baseUrl: XAI_GROK_OAUTH_BASE_URL,
90-
reasoning: false,
91-
input: ["text"],
92-
contextWindow: XAI_COMPOSER_CONTEXT_WINDOW,
93-
maxTokens: XAI_COMPOSER_MAX_TOKENS,
94-
cost: XAI_SUBSCRIPTION_COST_UNKNOWN,
95-
},
96-
] as const satisfies readonly XaiCatalogEntry[];
97-
9872
const XAI_MODEL_CATALOG = [
9973
{
10074
id: "grok-build-0.1",
@@ -262,8 +236,6 @@ function toModelDefinition(entry: XaiCatalogEntry): ModelDefinitionConfig {
262236
return {
263237
id: entry.id,
264238
name: entry.name,
265-
...(entry.api ? { api: entry.api } : {}),
266-
...(entry.baseUrl ? { baseUrl: entry.baseUrl } : {}),
267239
reasoning: entry.reasoning,
268240
input: entry.input ?? ["text"],
269241
cost: entry.cost,
@@ -292,17 +264,10 @@ export function buildXaiCatalogModels(): ModelDefinitionConfig[] {
292264
);
293265
}
294266

295-
export function buildXaiOAuthCatalogModels(): ModelDefinitionConfig[] {
296-
return [
297-
...XAI_OAUTH_MODEL_CATALOG.map((entry) => toModelDefinition(entry)),
298-
...buildXaiCatalogModels(),
299-
];
300-
}
301-
302267
export function resolveXaiCatalogEntry(modelId: string) {
303268
const trimmed = modelId.trim();
304269
const lower = normalizeXaiCatalogModelId(modelId);
305-
const exact = [...XAI_OAUTH_MODEL_CATALOG, ...XAI_MODEL_CATALOG].find(
270+
const exact = XAI_MODEL_CATALOG.find(
306271
(entry) => normalizeOptionalLowercaseString(entry.id) === lower,
307272
);
308273
if (exact) {

0 commit comments

Comments
 (0)