Skip to content

feat(xai): add realtime voice provider#99088

Closed
TheAngryPit wants to merge 11 commits into
openclaw:mainfrom
TheAngryPit:feat/xai-realtime-voice-provider
Closed

feat(xai): add realtime voice provider#99088
TheAngryPit wants to merge 11 commits into
openclaw:mainfrom
TheAngryPit:feat/xai-realtime-voice-provider

Conversation

@TheAngryPit

@TheAngryPit TheAngryPit commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Problem This Solves

OpenClaw Talk already has a realtime provider contract and gateway-relay path, but the bundled xAI plugin did not expose Grok Voice Agent as a native realtime voice provider.

Without this, users who want xAI/Grok voice in OpenClaw either cannot use it through Talk or have to fall back to a lower-quality STT -> model -> TTS loop. That is especially limiting for long-running assistants where supplier choice and realtime voice cost matter.

Closes #99087.

Why This Change Was Made

This adds one canonical bundled xAI realtime voice provider through the existing Talk gateway-relay contract instead of creating a separate voice stack.

The provider:

  • connects to xAI realtime voice (grok-voice-latest) over WebSocket;
  • registers through the existing registerRealtimeVoiceProvider seam;
  • exposes the provider through the xAI plugin manifest;
  • maps audio, transcript, tool-call, cancellation, reconnect, and close events into the existing Talk realtime bridge contract;
  • keeps secrets out of docs and evidence.

The follow-up commits address the ClawSweeper/Codex review findings:

  • interrupted assistant audio now sends response.cancel plus conversation.item.truncate;
  • relay playback cancellation preserves the assistant item until truncation is possible;
  • assistant transcript deltas are buffered so delta-only xAI completion still emits a final transcript;
  • xAI input_audio_transcription.updated events are treated as cumulative replacements and emitted once on completion;
  • provider-managed reconnects no longer replay initial onReady/greeting callbacks;
  • startup WebSocket errors remain terminal instead of leaving background retry loops;
  • reconnect continuity is now explicit opt-in through sessionResumption;
  • default sessions do not send session.resumption.enabled=true and fail closed on disconnect instead of silently starting a fresh provider conversation.

Dependency note: ws is intentionally moved into the xAI runtime dependency set because the new provider imports it at runtime. The repository dependency guard correctly requires maintainer/security approval for the exact head SHA or a maintainer-owned regeneration path. This PR does not try to bypass that policy.

User Impact

Users selecting xAI realtime Talk get native Grok voice replies through the existing OpenClaw Talk relay path rather than a synthetic STT/TTS loop.

Expected behavior:

  • Talk can select provider: "xai" with transport: "gateway-relay".
  • Gateway-relay clients can stream mic audio and receive Grok voice audio.
  • Barge-in cancels the active response and truncates only the currently interrupted assistant audio.
  • Completed assistant audio and previous-turn assistant items are not truncated on later turns.
  • Reconnect does not replay the initial greeting as a new user turn.
  • Provider-side xAI session replay/resumption is disabled by default and only enabled when sessionResumption: true is configured.

This is intentionally scoped to the xAI realtime voice provider and Talk relay integration. It does not weaken gateway auth and does not include secrets or production token values.

Evidence

Prior published validation head (superseded by the transcript checkpoint below):

Base: origin/main @ 5f3629e944
Head: 60a8060608

Prior published proof retained for continuity:

node scripts/run-vitest.mjs extensions/xai/realtime-voice-provider.test.ts
Result: passed, 1 file, 27 tests.

pnpm tsgo:prod
Result: passed.

pnpm check:test-types
Result: passed.

pnpm run test:extensions:package-boundary:compile
Result: passed, xai checked, extension package boundary check passed.

pnpm deps:shrinkwrap:check
Result: passed, all npm-shrinkwrap.json files current.

git diff --check origin/main...HEAD
Result: passed.

openclaw-autoreview --mode branch --base origin/main --engine codex --thinking high
Result: clean, no accepted/actionable findings. Overall: patch is correct (0.78).

Live Windows + WSL smoke proof already captured for the xAI gateway-relay voice path:

Environment: Windows + WSL.
Direct xAI bridge: connected to grok-voice-latest, ready=true, received audio deltas.
Isolated Gateway smoke: ran on non-production port 18790 because daily Gateway used 18789.
Talk runtime: talkMode=realtime, talkTransport=gateway-relay, talkProvider=xai.
Observed log events: output.audio.done, transcript.done, turn.cancelled, tool.call, tool.result, session.closed.
Operator result: Grok realtime voice replied through OpenClaw Talk; after correcting local speaker feedback, voice interaction was fast and worked correctly.

Workflow note:

For this branch I used the same review discipline we have been using on OpenClaw PRs: scoped local validation first, then openclaw-autoreview on the relevant diff/branch before publishing. Crabbox/Testbox remains the preferred broad proof surface for full CI-equivalent validation when available; it was not available in this local Windows pass, so the current-head proof above is focused local proof plus the existing live Windows+WSL smoke packet.

Agent Transcript

Sanitized worklog for PR #99088: xAI realtime voice provider

Goal

Add xAI realtime voice through OpenClaw's existing realtime voice provider and gateway relay contracts, while preserving the expanded official xAI provider now present on upstream main.

Continuity of work

This transcript is the latest checkpoint in a multi-day implementation and review effort, not a single-session change. Work on the PR was already active by July 3, 2026, when the branch had been rebased, provider tests were passing, and the external-contributor dependency gate was documented. Subsequent review rounds addressed barge-in truncation, assistant transcript buffering, reconnect behavior, delayed tool-result ownership, and relay cancellation. The July 10 pass rebased that accumulated work onto the official bundled xAI provider and re-proved the resulting integration.

Decisions and implementation

  • Verified that the official xAI provider still documents realtime voice as unavailable, so the PR remains a distinct feature rather than duplicate provider work.
  • Rebased the scoped realtime voice commit stack onto current upstream main.
  • Preserved current xAI model, media, transcription, OAuth, and provider documentation while adding only realtime voice registration and behavior.
  • Kept the runtime WebSocket dependency owned by the xAI extension. This intentional dependency graph change remains subject to maintainer/security approval.
  • Reconciled the gateway relay with current upstream barge-in semantics: provider-marked barge-in clears discard delayed tool results, while normal playback completion can still release them.
  • Preserved cancellation delivery when an in-flight OpenClaw consult is aborted during relay shutdown.

Failed-first proof and correction

  • The first focused run failed because relay shutdown sent chat.abort and closed the session without submitting the synthetic cancellation result to the provider.
  • The relay ownership path was corrected and the missing PCM test fixture was restored.
  • The repeated focused suite passed: 107 tests across the xAI provider, registration, relay, and realtime talk files.
  • git diff --check passed.
  • Scoped Codex autoreview returned no accepted or actionable findings and classified the patch as correct with confidence 0.82.

Remaining gate

  • The public PR still points to the previous head and therefore has not run CI on the rebased commits.
  • The public dependency guard remains expected until a maintainer/security reviewer authorizes the runtime dependency change.
  • One public review thread remains open; the rebased local implementation contains the requested delayed-result barge-in correction, but the thread must not be resolved until the new head is pushed and checks are re-read.

No secrets, credentials, raw prompts, private paths, or raw tool output are included.

@TheAngryPit
TheAngryPit requested a review from a team as a code owner July 2, 2026 13:18
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation app: web-ui App: web-ui extensions: xai size: XL labels Jul 2, 2026
@github-actions github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Dependency Guard

This PR changes dependency-related files. Maintainers should confirm these changes are intentional.

Changed files:

  • extensions/xai/package.json
  • pnpm-lock.yaml

Maintainer follow-up:

  • Review whether the dependency changes are intentional.
  • Inspect resolved package deltas when lockfile, shrinkwrap, or workspace dependency policy changes are present.
  • Treat package-lock.json and npm-shrinkwrap.json diffs as security-review surfaces.
  • Run pnpm deps:changes:report -- --base-ref origin/main --markdown /tmp/dependency-changes.md --json /tmp/dependency-changes.json locally for detailed release-style evidence.

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Dependency graph changes are blocked

OpenClaw does not accept dependency graph changes through PRs unless a repository admin or security explicitly authorizes the current head SHA. Dependency updates are generated internally by maintainers so external PRs cannot change the resolved graph.

Detected dependency graph changes:

  • pnpm-lock.yaml changed.
  • extensions/xai/package.json changed dependencies, devDependencies.

Auto-scrub was not attempted because this PR changes package manifest dependency graph fields:

  • extensions/xai/package.json changed dependencies, devDependencies.

Dependency graph changes must be reviewed by security or handled by maintainers internally. Please remove lockfile changes manually if they are not needed.

To remove lockfile changes, restore them from the target branch:

git fetch origin
git checkout 'origin/main' -- 'pnpm-lock.yaml'
git commit -m 'chore: remove dependency lockfile change'
git push

If this PR intentionally needs a dependency graph change, ask a repository admin or member of @openclaw/openclaw-secops to comment:

/allow-dependencies-change

The action will approve the current head SHA (da2467efd21a3980b52f35a24218a843b2b9c1fc) when it reruns. A later push requires a fresh approval.

@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 2, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 2, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 870d0cfa97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/xai/realtime-voice-provider.ts
@clawsweeper

clawsweeper Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 10, 2026, 3:20 PM ET / 19:20 UTC.

Summary
Adds a bundled xAI Grok realtime voice provider for Talk gateway relay, including WebSocket protocol handling, relay playback and tool-result coordination, provider registration, configuration documentation, focused tests, and a runtime ws dependency.

PR surface: Source +1269, Tests +1678, Docs +59, Config 0, Other 0. Total +3006 across 12 files.

Reproducibility: not applicable. as a bug reproduction; current main clearly lacks the requested provider, while the PR supplies source coverage and prior live smoke for the proposed capability.

Review metrics: 1 noteworthy metric.

  • Provider config surface: 10 optional settings added. Credentials, endpoint, model, voice, three VAD controls, reasoning effort, interruption, and resumption become an operator-facing contract requiring explicit acceptance.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #99087
Summary: This PR is the direct implementation candidate for the open xAI realtime voice feature issue; broader live-voice UI and batch TTS work are distinct or only partially overlapping.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Publish redacted real behavior proof from the exact rebased head.
  • Obtain maintainer approval for the bundled config surface and trusted authorization or regeneration of the runtime dependency graph.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The prior Windows and WSL smoke is relevant but predates exact head da2467e and later reconnect, relay shutdown, and delayed-result fixes; add redacted current-head live output, logs, or a recording, update the PR body, and allow automatic re-review or ask a maintainer to comment @clawsweeper re-review.

Mantis proof suggestion
A web UI Talk interaction can demonstrate the latest provider behavior, including visible transcript flow and barge-in, while returning redacted evidence. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis web UI chat proof: reproduce xAI gateway-relay Talk with a short voice turn, tool round trip, and barge-in, then return redacted transcript and interaction evidence.

Risk before merge

  • [P1] Merging adds ten optional xAI realtime provider settings, including provider-side session resumption; maintainers must accept and support this operator-facing contract.
  • [P1] The runtime ws dependency and resolved graph cannot land from this external branch without exact-head repository-admin or security authorization, or maintainer-owned regeneration.
  • [P1] The retained live smoke predates the exact head and does not prove the latest reconnect, cancellation, relay-close, and delayed tool-result behavior in a real setup.
  • [P1] The PR is behind current main and needs a refresh before final review; stale-base status alone is not evidence that the three-way merge would remove current-main behavior.

Maintainer options:

  1. Rebase, prove, and authorize (recommended)
    Refresh onto current main, add exact-head live evidence, and obtain trusted approval for both the public config contract and runtime dependency graph.
  2. Adopt through a maintainer branch
    A maintainer can preserve the provider implementation while narrowing config and regenerating dependency metadata through the trusted workflow.
  3. Pause bundled scope
    Keep the PR open or close it if maintainers do not want to own the new bundled provider and configuration surface.

Next step before merge

  • [P1] The next actions are contributor-owned exact-head proof and maintainer decisions on bundled product and configuration scope and the dependency graph; no narrow automated code repair is currently identified.

Maintainer decision needed

  • Question: Should OpenClaw bundle native xAI realtime voice now with this ten-setting provider config surface and runtime WebSocket dependency?
  • Rationale: The implementation fits existing architecture, but accepting a new bundled provider, durable operator-facing configuration contract, and shipped dependency is a product and ownership decision that automated review cannot make.
  • Likely owner: Peter Steinberger — His merged history introduced and refined the plugin-owned realtime voice architecture that this product decision extends.
  • Options:
    • Bundle the provider (recommended): Accept the gateway-relay provider after rebase, exact-head real behavior proof, and trusted dependency authorization.
    • Narrow the public config: Keep the provider but reduce or defer optional settings such as session resumption before establishing the supported contract.
    • Do not bundle it: Decline the core feature and leave xAI realtime voice to an external plugin if bundled provider scope should remain smaller.

Security
Needs attention: No credential leak or authorization bypass was found, but the intentional shipped ws dependency graph change requires trusted exact-head supply-chain authorization.

Review details

Best possible solution:

Rebase onto current main, retain the plugin-owned xAI bridge and provider-neutral relay clear semantics, publish redacted exact-head Talk proof covering audio, transcripts, a tool round trip, barge-in, and close or resumed reconnect behavior, then obtain maintainer approval for the config surface and trusted authorization or regeneration of the dependency graph.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug reproduction; current main clearly lacks the requested provider, while the PR supplies source coverage and prior live smoke for the proposed capability.

Is this the best way to solve the issue?

Yes on architecture: the existing xAI plugin plus generic Talk gateway-relay seam is the narrowest maintainable ownership boundary; merge still requires exact-head proof and maintainer acceptance of the public config and dependency contract.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e1934d968e43.

Label changes

Label changes:

  • add merge-risk: 🚨 compatibility: The PR adds a persistent provider configuration surface and provider-side resumption behavior requiring fresh-install, upgrade, documentation, and operator-contract review.

Label justifications:

  • P2: This is a substantial optional provider feature with limited blast radius rather than an urgent regression.
  • merge-risk: 🚨 compatibility: The PR adds a persistent provider configuration surface and provider-side resumption behavior requiring fresh-install, upgrade, documentation, and operator-contract review.
  • merge-risk: 🚨 security-boundary: The PR changes the shipped xAI runtime dependency graph and requires trusted exact-head dependency authorization.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The prior Windows and WSL smoke is relevant but predates exact head da2467e and later reconnect, relay shutdown, and delayed-result fixes; add redacted current-head live output, logs, or a recording, update the PR body, and allow automatic re-review or ask a maintainer to comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +1269, Tests +1678, Docs +59, Config 0, Other 0. Total +3006 across 12 files.

View PR surface stats
Area Files Added Removed Net
Source 5 1277 8 +1269
Tests 4 1678 0 +1678
Docs 1 79 20 +59
Config 1 3 3 0
Generated 0 0 0 0
Other 1 3 3 0
Total 12 3040 34 +3006

Security concerns:

  • [medium] Authorize the exact runtime dependency graph — extensions/xai/package.json:9
    The xAI plugin imports ws on its production path and moves it into shipped dependencies; repository policy requires a repository admin or security reviewer to approve the exact head or regenerate the dependency files internally.
    Confidence: 0.99

What I checked:

  • Feature remains absent: Current main explicitly documents xAI realtime voice as not exposed and not registered, so the central capability is still necessary. Public docs: docs/providers/xai.md. (docs/providers/xai.md:142, e1934d968e43)
  • Correct owner boundary: The implementation keeps xAI credentials, protocol, model and voice settings, and bridge lifecycle in the bundled xAI plugin and registers through the existing generic realtime voice provider API. (extensions/xai/realtime-voice-provider.ts:1126, da2467efd21a)
  • Protocol invariants implemented: The bridge implements opt-in resumption with conversation identity, queues reconnect-window input and tool results, waits for tool outputs and playback before creating the next response, finalizes transcripts, and truncates interrupted assistant audio; these match the official xAI Voice Agent contracts linked in the review discussion. (extensions/xai/realtime-voice-provider.ts:364, da2467efd21a)
  • Provider-neutral relay behavior: The Gateway carries an audio-clear reason to the browser so barge-in clears discard delayed tool results while ordinary playback completion can release them. (src/gateway/talk-realtime-relay.ts:684, da2467efd21a)
  • Real proof is not exact-head: The PR body retains a positive Windows and WSL live smoke from an earlier checkpoint, while later reconnect, relay shutdown, cancellation, delayed-result, and assertion fixes have only focused test evidence. (da2467efd21a)
  • Dependency graph is gated: The plugin moves ws into its shipped runtime dependencies and changes the root lockfile; the live dependency guard requires repository-admin or security approval for the exact head, or trusted maintainer regeneration. (extensions/xai/package.json:7, da2467efd21a)

Likely related people:

  • Peter Steinberger: Introduced the plugin-owned realtime voice provider boundary and carried major OpenAI realtime provider cleanup and deduplication work that defines the sibling contract this PR follows. (role: realtime voice architecture contributor; confidence: high; commits: a23ab9b906dc, 7cd40ad5654a, 6dfdc92bd4c6; files: extensions/openai/realtime-voice-provider.ts, src/gateway/talk-realtime-relay.ts, ui/src/pages/chat/realtime-talk-gateway-relay.ts)
  • Vincent Koc: Has extensive merged history on xAI provider helpers, shared replay and tool contracts, and provider SDK alignment, making him a strong reviewer for the xAI ownership and shared-contract aspects. (role: xAI and provider-family contributor; confidence: high; commits: 9c42e6424d25, 4c15f1310bd5, 65842aabada0; files: extensions/xai/index.ts, docs/providers/xai.md, src/plugin-sdk/realtime-voice.ts)
  • Tak Hoffman: Introduced the proxy-capture stack used by this provider's WebSocket diagnostics, so he is relevant to the debug-proxy and capture integration rather than the product decision. (role: adjacent runtime contributor; confidence: medium; commits: 958c34e82cdb; files: extensions/xai/realtime-voice-provider.ts, src/plugin-sdk/proxy-capture.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (11 earlier review cycles; latest 8 shown)
  • reviewed 2026-07-05T00:30:56.331Z sha 1017190 :: found issues before merge. :: [P2] Preserve xAI context across reconnects
  • reviewed 2026-07-06T10:03:23.118Z sha 690f3f8 :: found issues before merge. :: [P2] Keep the websocket reference non-null before sending | [P2] Defer xAI response.create until playback drains
  • reviewed 2026-07-06T11:18:53.078Z sha c8c51de :: needs changes before merge. :: [P2] Delay xAI tool follow-ups until relay playback drains
  • reviewed 2026-07-06T15:31:18.356Z sha 7be6ed5 :: needs real behavior proof before merge. :: [P2] Use a browser-compatible timer type | [P3] Remove the spread copies from delayed-result loops
  • reviewed 2026-07-06T19:58:26.335Z sha 0b93e89 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-08T18:21:20.652Z sha 9cafc3c :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-08T18:30:15.956Z sha 9cafc3c :: needs real behavior proof before merge. :: [P2] Discard delayed results on provider barge-in clears
  • reviewed 2026-07-10T19:08:08.766Z sha da2467e :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot mentioned this pull request Jul 2, 2026
Closed
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jul 2, 2026
@TheAngryPit
TheAngryPit force-pushed the feat/xai-realtime-voice-provider branch from 870d0cf to b6f54f2 Compare July 2, 2026 14:39
@TheAngryPit

TheAngryPit commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up pushed for the ClawSweeper/Codex review finding.

What changed:

  • Rebased the branch on current upstream/main (4614449d9859).
  • Fixed the xAI realtime voice barge-in path so interrupted assistant audio sends response.cancel plus conversation.item.truncate.
  • Covered the relay-specific edge case where marks are acknowledged immediately but the response is still active.
  • Added negative coverage so completed assistant audio and previous-turn assistant items are not truncated on later turns.
  • Updated docs/providers/xai.md so the documented xAI Talk barge-in behavior matches the implementation.

Current-head proof:

node scripts/run-vitest.mjs run extensions/xai/realtime-voice-provider.test.ts extensions/xai/index.test.ts
Result: passed, 2 files, 34 tests.

node scripts/run-vitest.mjs run ui/src/ui/realtime-talk.test.ts ui/src/ui/realtime-talk-gateway-relay.test.ts
Result: passed, 2 files, 20 tests.

focused oxlint on xAI provider/test/index/docs
Result: passed.

git diff --check upstream/main...HEAD
Result: passed.

openclaw-autoreview --mode branch --base upstream/main --engine codex --model gpt-5.5 --thinking high
Result: clean, no accepted/actionable findings. Overall: patch is correct (0.84).

Dependency guard note: the remaining dependency-graph block is expected and maintainer/security-owned because ws is intentionally a runtime dependency for this provider. I did not try to bypass that policy.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 2, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 2, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6f54f2801

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/xai/realtime-voice-provider.ts Outdated
@TheAngryPit

TheAngryPit commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up pushed at current head 83dc00fe3f.

This addresses the remaining ClawSweeper/Codex feedback together:

  • xAI assistant transcript deltas are now buffered and flushed as a final assistant transcript when the xAI done event is only a completion marker.
  • The barge-in fix now preserves the assistant item/timestamp through the no-op server-VAD fallback, so a later relay cancelOutput with audioPlaybackActive=true can still send conversation.item.truncate instead of losing the item id.
  • Added focused regression coverage for both delta-only transcript completion and the post-response.done playback-cancel race.

Fresh current-head validation:

node scripts/run-vitest.mjs run extensions/xai/realtime-voice-provider.test.ts extensions/xai/index.test.ts
Result: passed, 2 files, 36 tests.

node scripts/run-vitest.mjs run ui/src/ui/realtime-talk.test.ts ui/src/ui/realtime-talk-gateway-relay.test.ts
Result: passed, 2 files, 20 tests.

node scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.extensions.json extensions/xai/realtime-voice-provider.ts extensions/xai/realtime-voice-provider.test.ts extensions/xai/index.ts extensions/xai/index.test.ts
Result: passed.

git diff --check upstream/main...HEAD
Result: passed.

openclaw-autoreview --mode branch --base upstream/main --engine codex --model gpt-5.5 --thinking high
Result: clean, no accepted/actionable findings. Overall: patch is correct (0.78).

The dependency guard remains expected and maintainer/security-owned for the current head SHA because this PR intentionally changes the xAI runtime dependency metadata for ws.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 2, 2026
@steipete steipete mentioned this pull request Jul 10, 2026
Closed
@TheAngryPit
TheAngryPit force-pushed the feat/xai-realtime-voice-provider branch from 9cafc3c to e484c09 Compare July 10, 2026 18:24
@openclaw-barnacle openclaw-barnacle Bot added the gateway Gateway runtime label Jul 10, 2026
@TheAngryPit TheAngryPit mentioned this pull request Jul 10, 2026
Closed
@clawsweeper

clawsweeper Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat(xai): add realtime voice provider This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 10, 2026
@steipete steipete self-assigned this Jul 13, 2026
This was referenced Jul 13, 2026
Closed
Merged
@steipete

steipete commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Superseded by #106267, which preserves @TheAngryPit’s contributor credit and landed the canonical xAI realtime Talk implementation in 6c5084f1adb.

The landed version keeps xAI auth, session configuration, events, tools, server VAD, reconnect, and resumption in the xAI plugin while core and clients carry only generic realtime relay contracts. Exact-head release gate 29248320992 passed all 117 jobs, and authenticated live proof covered real audio, tool results, barge-in, reconnect, and resumed context.

Closing this older ancestry-noisy branch as superseded. Thank you, Vitor, for the original implementation and the sustained review iterations.

@steipete steipete closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

@steipete steipete

Labels

app: web-ui App: web-ui dependencies-changed PR changes dependency-related files docs Improvements or additions to documentation extensions: xai gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P2 Normal backlog priority with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add native xAI realtime voice provider for Talk gateway-relay

3 participants

@TheAngryPit @steipete @CassieMei