feat(xai): add realtime voice provider by steipete · Pull Request #106267 · openclaw/openclaw

steipete · 2026-07-13T10:12:46Z

Supersedes #99088, the stale duplicate #91308, and the ancestry-noisy replacement #106150. This maintainer-owned replacement preserves contributor credit for @TheAngryPit after the original fork branch could not accept the verified prepared tree.

What Problem This Solves

OpenClaw Talk has a realtime provider contract and gateway-relay path, but the bundled xAI plugin does not expose Grok Voice Agent as a native realtime voice provider. Users therefore cannot use xAI voice through Talk without falling back to a slower STT-model-TTS loop.

Why This Change Was Made

This adds one canonical bundled xAI realtime provider through the existing Talk gateway-relay contract. xAI-specific WebSocket auth, session configuration, transcripts, tools, server-VAD interruption, and optional resumption stay inside the xAI plugin; core, gateway, UI, iOS, and Android carry only generic typed playback marks and interruption state.

The maintainer repair also splits the original provider monolith into bridge, config, event, and protocol owners; rejects unsupported generic response/interruption modes instead of silently diverging from xAI semantics; and truncates provider history only to the relay's acknowledged playback boundary. The implementation was checked against the current xAI Voice Agent guide and Voice API reference.

Final hardening constrains xAI reasoning effort to the provider's supported high/none values, regenerates the Android method catalog for playback acknowledgements, and holds final tool results behind every outstanding playback-cancellation request. Cancellation failure now closes the relay instead of releasing a result against unconfirmed provider state.

ws is intentionally added to the xAI plugin runtime dependency set because the provider imports it at runtime. The exact-head dependency guard therefore requires maintainer authorization.

User Impact

Users can select xAI realtime Talk with provider: "xai" and transport: "gateway-relay", stream microphone audio, receive native Grok voice audio, invoke tools, interrupt queued playback without corrupting conversation history, and opt into provider session resumption.

Evidence

Final branch head: 72e064280b8005e79198e90965c452632f64ac4d (branch history preserves co-authorship by Vitor Cepeda Lopes). This is the clean 43-file comparison after merging current main; the earlier #106150 append-only sync retained stale ancestry and therefore exposed an unrelated 560-file comparison. The feature patch remains the reviewed implementation, with the release-owned CHANGELOG.md entry removed as required by the current prepare gate. Generated native catalogs were aligned with the integrated source tree; final main syncs include the current Control UI route, export-baseline, and event-envelope gate repairs.

Exact code-head Blacksmith Testbox protocol, focused test, lint/format, and production build proof: run 29239566954. Feature tree a4fd07a91dffd456c5e61185ccc549cf08b12fe2 then passed native/UI/protocol generators, 242 focused assertions, full pnpm check:changed, and full pnpm build in Testbox. Final integrated tree 0d2284dc5b2250ec3f8e0c6530a1cceac35fa5e9 has native inventory and JSON/Swift/Kotlin protocol generators idempotent; exhaustive exact-head release gate 29248320992 passed all 117 jobs, with only the optional timings summary skipped. The clean replacement stack has git diff --check clean. Earlier branch-wide changed gate: run 29235425605. Content-identical feature/mobile/live proof: run 29236039421.

pnpm test extensions/xai/index.test.ts extensions/xai/realtime-voice-provider.test.ts extensions/openai/realtime-voice-provider.test.ts extensions/google/realtime-voice-provider.test.ts
Result: 184 passed.

pnpm test src/talk/session-runtime.test.ts
Result: 13 passed.

pnpm test src/gateway/talk-realtime-relay.test.ts src/gateway/server-methods/talk.test.ts src/gateway/method-scopes.test.ts src/gateway/server-methods-list.test.ts
Result: 760 passed.

pnpm test packages/gateway-protocol
Result: 205 passed.

pnpm test ui/src/pages/chat/realtime-talk-gateway-relay.test.ts ui/src/pages/chat/realtime-talk.test.ts
Result: 35 passed.

pnpm test test/scripts/test-live-shard.test.ts
Result: 20 passed.

pnpm protocol:gen && pnpm protocol:gen:swift
Result: generated JSON and Swift protocol hashes matched the reviewed tree.

pnpm protocol:check
Result: JSON, Swift, and Kotlin generated protocol files are exact code-head idempotent.

pnpm test extensions/xai/realtime-voice-provider.test.ts ui/src/pages/chat/realtime-talk-gateway-relay.test.ts src/gateway/talk-realtime-relay.test.ts
Result: 242 exact code-head assertions passed, including invalid xAI reasoning effort, provider clear, overlapping cancellation, and cancellation-failure regressions.

pnpm check:changed
Result: the feature head passed format, shrinkwrap, Plugin SDK API baseline, all-project typecheck, full lint, and import cycles. `pnpm plugin-sdk:surface:check` passes and the final repair touches no SDK surface.

pnpm build
Result: exact code-head production build passed, including plugin assets, tsdown, SDK export checks, and Control UI.

OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/xai/xai.live.test.ts
Result: 8 passed, 2 optional video tests skipped. The 17.1-second realtime scenario used synthesized PCM input and verified real provider audio, tool/result audio, queued-playback barge-in, provider truncation acknowledgement, forced reconnect, resumed-context marker recall, and no redundant response.cancel.

Content-identical mobile proof also passed iOS relay tests, Swift format/lint, the Android playback-barrier regression, and Android ktlint. Fresh full-patch autoreview is clean after the final overlap-race repair; confidence 0.90.

Co-authored-by: Vitor Cepeda Lopes <[email protected]>

github-actions · 2026-07-13T10:13:07Z

Dependency Guard

This PR changes dependency-related files. Maintainers should confirm these changes are intentional.

Changed files:

extensions/xai/package.json
pnpm-lock.yaml

Maintainer follow-up:

Review whether the dependency changes are intentional.
Inspect resolved package deltas when lockfile, shrinkwrap, or workspace dependency policy changes are present.
Treat package-lock.json and npm-shrinkwrap.json diffs as security-review surfaces.
Run pnpm deps:changes:report -- --base-ref origin/main --markdown /tmp/dependency-changes.md --json /tmp/dependency-changes.json locally for detailed release-style evidence.

github-actions · 2026-07-13T10:13:07Z

Dependency graph changes noted

This PR includes dependency graph changes. The dependency guard is informational because the PR author is a repository admin or a member of @openclaw/openclaw-secops.

Current SHA: 72e064280b8005e79198e90965c452632f64ac4d
Trusted actor: @steipete
Trusted role: pull request author; openclaw-secops

Security review is still recommended before merge when the dependency graph change is intentional.

steipete · 2026-07-13T10:13:19Z

/allow-dependencies-change

steipete · 2026-07-13T10:13:35Z

Land-ready feature proof for clean replacement head 72e064280b8005e79198e90965c452632f64ac4d. This is the xAI feature stack merged with current main as a clean 43-file comparison; git diff --check is clean, and release-note context plus contributor credit remain in the PR body.

Blacksmith Testbox run 29239566954: exact code-head SHA assertion, touched-file format/oxlint, pnpm protocol:check, 242 focused assertions, and full pnpm build all passed.
Authenticated xAI live proof: 8 passed, 2 optional video cases skipped. The real 17.1-second realtime case exercised audio, tool/result audio, queued-playback barge-in, provider truncation acknowledgement, reconnect, and resumed-context recall.
Mobile proof: Swift format/lint, iOS relay tests, Android playback-barrier regression, and Android ktlint passed on the feature tree. The final generated Android method catalog is idempotent in the exact-head protocol check.
Fresh full-patch autoreview is clean after repairing the delayed-result, xAI reasoning-effort, and overlapping-cancellation findings.
Feature Testbox tree a4fd07a91dffd456c5e61185ccc549cf08b12fe2 passed native/UI i18n and protocol generators, 242 focused assertions, full pnpm check:changed (types, lint, import cycles, SDK and guards), and full pnpm build. Final integrated tree 0d2284dc5b2250ec3f8e0c6530a1cceac35fa5e9 has native inventory and JSON/Swift/Kotlin protocol generators idempotent. Exhaustive exact-head release gate 29248320992 passed all 117 jobs; only the optional timings summary skipped.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c343bd26d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T10:22:01Z

+    this.markQueue.push(markName);
+    this.config.onMark?.(markName);


Handle missing mark acknowledgers

When this bridge is used without an onMark callback (allowed by RealtimeVoiceBridgeCallbacks, and also what the runtime's markStrategy: "ignore" does), sendMark() still appends every audio delta to markQueue but no one can call acknowledgeMark(). Because requestResponseCreate() refuses to create the next response while markQueue.length > 0, any later text turn or tool result stays stuck in responseCreatePending, so non-mark transports hang after the first audio response.

Useful? React with 👍 / 👎.

clawsweeper · 2026-07-13T11:00:14Z

Codex review: stale review; fresh review needed.

Summary
The latest durable ClawSweeper review was for head f456072b96c9c8efb4240de87c626f4b95b28bdc, but the PR head is now 72e064280b8005e79198e90965c452632f64ac4d. Its old verdict and PR readiness labels are no longer current.

Next step
Run or wait for a fresh ClawSweeper review on the current PR head.

Review history (3 earlier review cycles)

reviewed 2026-07-13T10:58:10.262Z sha 70d29d2 :: found issues before merge. :: [P2] Drain marks when no acknowledger is installed
reviewed 2026-07-13T11:13:42.382Z sha 6c330be :: found issues before merge. :: [P2] Drain marks when no acknowledger is installed
reviewed 2026-07-13T11:28:27.197Z sha dd6b5f4 :: found issues before merge. :: [P2] Derive truncation from acknowledged playback

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c330bea56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T11:14:28Z

+    const audioEndMs = shouldInterruptProvider
+      ? Math.max(
+          0,
+          responseStartTimestamp === null
+            ? this.latestMediaTimestamp
+            : this.latestMediaTimestamp - responseStartTimestamp,
+        )


Derive truncation from playback, not input timestamps

When native clients suppress microphone frames during assistant output (Android does this in TalkModeManager.kt:1064, and iOS does it on the built-in-speaker path in RealtimeTalkRelaySession.swift:1052-1059), latestMediaTimestamp stops advancing right when the response starts. A later cancelOutput/barge-in can therefore send conversation.item.truncate with audio_end_ms near 0 even after the user heard several seconds of audio, causing xAI to remove already-played assistant content from the provider history and making follow-up turns lose context. The truncation point needs to come from the transport playback boundary/mark acknowledgements, or from an explicit playback offset passed with the cancel request, rather than from the last appended input frame timestamp.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd6b5f498b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T11:29:45Z

+      this.scheduleDelayedToolResult({ callId, result, ...(options ? { options } : {}) });
+      return;


Return delayed tool-result submission promises

When a final consult result arrives while assistant audio is still queued, this branch schedules the actual talk.session.submitToolResult for later but returns a resolved promise immediately. submitRealtimeTalkConsult treats that as provider submission complete, switches the UI back to listening, and drops the call's abort tracking even though the provider is still waiting for the function output; if the delayed submission later fails or is held behind cancellation, the UI exposes a new listening turn during an unresolved tool call. The delayed path should resolve/reject only when sendToolResultNow actually runs.

Useful? React with 👍 / 👎.

steipete · 2026-07-13T12:17:28Z

Merged via squash.

Prepared head SHA: 72e064280b8005e79198e90965c452632f64ac4d
Landed commit: 6c5084f1adb39088c19788b8fbe200b9c3c382d9

* feat(xai): add realtime voice provider Co-authored-by: Vitor Cepeda Lopes <[email protected]> * fix(xai): satisfy realtime voice validation * fix(xai): harden realtime relay boundaries * chore: drop release-owned changelog edit * chore(i18n): refresh native voice catalogs * chore(i18n): refresh Control UI raw-copy baseline * chore(plugin-sdk): refresh API baseline hashes * chore(i18n): use canonical main catalogs * chore(i18n): refresh realtime voice source inventory * chore: refresh xai voice landing head [skip ci] * chore: refresh integrated protocol artifacts [skip ci] --------- Co-authored-by: Vitor Cepeda Lopes <[email protected]>

steipete and others added 4 commits July 13, 2026 11:10

feat(xai): add realtime voice provider

9beee9a

Co-authored-by: Vitor Cepeda Lopes <[email protected]>

fix(xai): satisfy realtime voice validation

ebf9ed6

fix(xai): harden realtime relay boundaries

1579f2a

chore: drop release-owned changelog edit

c343bd2

steipete requested a review from a team as a code owner July 13, 2026 10:12

github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 13, 2026

openclaw-barnacle Bot added docs Improvements or additions to documentation app: android App: android app: ios App: ios app: web-ui App: web-ui gateway Gateway runtime extensions: openai extensions: google extensions: xai size: XL maintainer Maintainer-authored PR labels Jul 13, 2026

steipete mentioned this pull request Jul 13, 2026

feat(xai): add realtime voice provider #106150

Closed

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

steipete added 8 commits July 13, 2026 11:24

chore(i18n): refresh native voice catalogs

f7d1b2f

chore: merge latest main for xAI voice landing

c36eb80

chore(i18n): refresh Control UI raw-copy baseline

fafa3e6

chore(plugin-sdk): refresh API baseline hashes

5401bd5

chore: merge latest main CI fixes

b8b1b90

chore(i18n): use canonical main catalogs

70d29d2

chore: merge main typecheck fix

31cf1a5

chore(i18n): refresh realtime voice source inventory

6c330be

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

steipete added 2 commits July 13, 2026 12:16

chore: refresh xai voice landing head [skip ci]

2d87730

chore: sync current main for xai voice landing [skip ci]

dd6b5f4

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

steipete added 2 commits July 13, 2026 12:39

chore: refresh integrated protocol artifacts [skip ci]

bb324f9

chore: sync repaired main for xai voice landing [skip ci]

12b5982

steipete added 2 commits July 13, 2026 12:53

chore: sync control ui gate repair [skip ci]

f456072

chore: sync chat type gate repair [skip ci]

72e0642

steipete merged commit 6c5084f into main Jul 13, 2026
141 of 142 checks passed

steipete deleted the codex/pr-99088-xai-realtime-voice-land-v2 branch July 13, 2026 12:17

This was referenced Jul 13, 2026

feat(xai): add realtime voice provider #99088

Closed

feat(xai): add realtime voice provider #91308

Closed

Add native xAI realtime voice provider for Talk gateway-relay #99087

Closed

github-actions Bot mentioned this pull request Jul 13, 2026

📡 Upstream Digest — 2026-07-13 12:26 UTC curtismercier/openclaw-mods#1281

Open

teaspoonai mentioned this pull request Jul 13, 2026

feat: advertise realtime voice provider voices in talk.catalog #106023

Open

zenglingbiao mentioned this pull request Jul 26, 2026

fix(xai): reject malformed base64 audio in realtime voice deltas #114046

Open

		this.markQueue.push(markName);
		this.config.onMark?.(markName);

		this.scheduleDelayedToolResult({ callId, result, ...(options ? { options } : {}) });
		return;

Uh oh!

Uh oh!

Conversation

steipete commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

github-actions Bot commented Jul 13, 2026

Dependency Guard

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency graph changes noted

Uh oh!

steipete commented Jul 13, 2026

Uh oh!

steipete commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

clawsweeper Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steipete commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

steipete commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading

steipete commented Jul 13, 2026 •

edited

Loading

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading