Skip to content

fix: route iOS OpenAI realtime Talk through WebRTC#98563

Merged
steipete merged 10 commits into
openclaw:mainfrom
PollyBot13:fix-ios-talk-realtime-routing-reviewed
Jul 2, 2026
Merged

fix: route iOS OpenAI realtime Talk through WebRTC#98563
steipete merged 10 commits into
openclaw:mainfrom
PollyBot13:fix-ios-talk-realtime-routing-reviewed

Conversation

@PollyBot13

@PollyBot13 PollyBot13 commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Related: #91007

What Problem This Solves

Fixes an issue where iOS users trying to use OpenAI Realtime Talk could see the app enter the Gateway Relay path even though the Gateway advertised a WebRTC realtime configuration.

Why This Change Was Made

The Gateway already carries the requested realtime transport. This change makes the iOS Talk route honor OpenAI Realtime configs with transport: "webrtc" by using the existing native WebRTC session path instead of mapping them to Gateway Relay.

User Impact

iOS Talk can use the intended Native WebRTC path for OpenAI Realtime configurations that request WebRTC, while existing relay/provider-websocket behavior remains available for configurations that explicitly need it.

Evidence

  • External screenshot/video proof media was removed after merge for privacy.
  • Physical iPhone proof before merge showed GPT Realtime 2.0 on Native WebRTC, Talk Ready -> Listening on Native WebRTC, and a completed user prompt plus app response.
  • Maintainer repair landed on exact head 059023e7800350bd59d922b81465c5004f90509d; branch-wide review and hosted/local validation were completed before merge.
  • TalkModeManagerTests: 32 passed.
  • TalkConfigParsingTests: 7 passed.
  • iOS simulator build passed.
  • SwiftFormat, SwiftLint, git diff --check, pnpm native:i18n:check, docs formatting, MDX syntax check, and docs link audit passed.

AI-assisted: yes. The implementation and review were done with Codex, with local review findings addressed before publication.

@PollyBot13

PollyBot13 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Current head: fdf6532

All currently reported checks are passing or intentionally skipped; PR body is updated with the focused transport-routing evidence and validation.

@clawsweeper

clawsweeper Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper

clawsweeper Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge. Reviewed July 2, 2026, 3:40 PM ET / 19:40 UTC.

Summary
This PR routes eligible iOS OpenAI Realtime Talk configs to native WebRTC, preserves Gateway-owned realtime routes, updates audio/session recovery, adds Talk/Gateway tests and docs, syncs native i18n metadata, and includes a reply-init test fixture atomic-write hunk.

PR surface: Source +30, Tests +117, Docs +2, Other +725. Total +874 across 12 files.

Reproducibility: yes. by source inspection and linked live evidence: current main resolves the iOS OpenAI Realtime selection to Gateway Relay, while the canonical issue reports the matching relay/session.error/Unknown Talk session failure. I did not run a local iPhone repro in this read-only review.

Review metrics: 1 noteworthy metric.

  • iOS realtime route ownership: 1 default route changed, 4 Gateway-owned cases preserved. The PR intentionally changes eligible OpenAI WebRTC routing while preserving explicit relay, provider-websocket, Azure, and non-OpenAI realtime ownership.

Stored data model
Persistent data-model change detected: serialized state: src/config/sessions/session-accessor.reply-init-concurrency.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #91007
Summary: This PR is a candidate fix for the OpenAI native WebRTC routing slice of the canonical iOS Talk session issue; the current merge conflict is from an unrelated fixture hunk already merged separately.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster ✨ media proof bonus
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Risk before merge

  • [P1] Eligible existing iOS OpenAI realtime configs that omit transport or explicitly request WebRTC will move from Gateway Relay to native WebRTC after upgrade.
  • [P1] Credential and session ownership changes from a Gateway relay session to a Gateway-minted native WebRTC client session for eligible OpenAI configs; explicit relay, provider-websocket, Azure, and non-OpenAI cases must stay preserved.
  • [P1] Current head is merge-conflicting with current main because the reply-init fixture hunk was separately merged in fix(ci): session concurrency test flakes during child handshake #99212.

Maintainer options:

  1. Resolve Conflict And Accept Native WebRTC Default (recommended)
    Drop or align the superseded reply-init fixture hunk from fix(ci): session concurrency test flakes during child handshake #99212, then land the guarded native WebRTC default if maintainers accept the upgrade behavior.
  2. Keep Relay As The Default
    If maintainers reject the omitted-transport compatibility shift, require a small change before merge so only explicit transport=webrtc moves iOS to native WebRTC.
  3. Pause For Upgrade Proof
    If the compatibility impact remains uncertain, pause landing until an existing relay-backed iOS setup is tested across upgrade with explicit transport selection.

Next step before merge

  • [P2] A narrow automated repair can remove or align the superseded fixture hunk that blocks merging; maintainer approval is still needed for the compatibility/auth-provider route decision.

Security
Cleared: No concrete security or supply-chain regression was found; the auth-sensitive route still uses Gateway-minted client sessions and keeps source SecretRefs redacted.

Review findings

  • [P2] Drop the superseded session fixture hunk before merge — src/config/sessions/session-accessor.reply-init-concurrency.test.ts:87-89
Review details

Best possible solution:

Rebase the branch to remove or align the superseded fixture hunk, then land the iOS WebRTC routing fix once maintainers accept the default-route compatibility and auth-provider ownership shift.

Do we have a high-confidence way to reproduce the issue?

Yes, by source inspection and linked live evidence: current main resolves the iOS OpenAI Realtime selection to Gateway Relay, while the canonical issue reports the matching relay/session.error/Unknown Talk session failure. I did not run a local iPhone repro in this read-only review.

Is this the best way to solve the issue?

Yes for the central bug: resolving native WebRTC in the iOS routing layer is the right owner boundary because the Gateway already exposes client-owned WebRTC sessions. The branch still needs a conflict repair and maintainer acceptance of the default-route compatibility shift before merge.

Full review comments:

  • [P2] Drop the superseded session fixture hunk before merge — src/config/sessions/session-accessor.reply-init-concurrency.test.ts:87-89
    This PR now conflicts with current main because the same reply-init fixture atomic-write fix was merged separately in fix(ci): session concurrency test flakes during child handshake #99212 with a slightly different hunk. Rebase and remove or align this unrelated test-file change so the iOS Talk fix can merge cleanly.
    Confidence: 0.91

Overall correctness: patch is incorrect
Overall confidence: 0.87

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a004e18b4bbd.

Label changes

Label changes:

  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (recording): Sufficient: inspected physical-iPhone proof frames show Native WebRTC Ready/Listening states and the linked transcript records a completed audible prompt/response; the prepared manifest notes ffprobe was unavailable for the video file.
  • remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.
  • remove status: 👀 ready for maintainer look: Current PR status label is status: ⏳ waiting on author.

Label justifications:

  • P1: The PR targets a broken iOS OpenAI Realtime Talk workflow that can make voice interaction unusable on a connected iPhone.
  • merge-risk: 🚨 compatibility: Existing eligible iOS OpenAI realtime configs can change runtime path from Gateway Relay to native WebRTC after upgrade.
  • merge-risk: 🚨 auth-provider: The route change shifts credential and session ownership from Gateway relay startup to Gateway-minted native WebRTC client sessions.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (recording): Sufficient: inspected physical-iPhone proof frames show Native WebRTC Ready/Listening states and the linked transcript records a completed audible prompt/response; the prepared manifest notes ffprobe was unavailable for the video file.
  • proof: sufficient: Contributor real behavior proof is sufficient. Sufficient: inspected physical-iPhone proof frames show Native WebRTC Ready/Listening states and the linked transcript records a completed audible prompt/response; the prepared manifest notes ffprobe was unavailable for the video file.
  • proof: 🎥 video: Contributor real behavior proof includes video or recording evidence. Sufficient: inspected physical-iPhone proof frames show Native WebRTC Ready/Listening states and the linked transcript records a completed audible prompt/response; the prepared manifest notes ffprobe was unavailable for the video file.
Evidence reviewed

PR surface:

Source +30, Tests +117, Docs +2, Other +725. Total +874 across 12 files.

View PR surface stats
Area Files Added Removed Net
Source 1 35 5 +30
Tests 2 119 2 +117
Docs 2 6 4 +2
Config 0 0 0 0
Generated 0 0 0 0
Other 7 852 127 +725
Total 12 1012 138 +874

Acceptance criteria:

  • [P1] git diff --check.
  • [P1] pnpm native:i18n:check.
  • [P1] xcodebuild -project apps/ios/OpenClaw.xcodeproj -scheme OpenClaw -destination 'platform=iOS Simulator,name=iPhone 17,OS=26.5' -only-testing:OpenClawTests/TalkModeManagerTests test.
  • [P1] xcodebuild test -project apps/ios/OpenClaw.xcodeproj -scheme OpenClawLogicTests -destination 'platform=iOS Simulator,name=iPhone 17,OS=26.5' CODE_SIGNING_ALLOWED=NO ARCHS=arm64 ONLY_ACTIVE_ARCH=YES VALID_ARCHS=arm64.

What I checked:

  • Repository policy read: Root AGENTS.md and scoped iOS, Gateway, Gateway server-methods, docs, and scripts guides were read fully; their compatibility/auth-provider and scoped review guidance shaped the verdict. (AGENTS.md:1, a004e18b4bbd)
  • Current main still routes OpenAI selection to relay: Current main resolves the iOS OpenAI Realtime picker to realtimeRelay, matching the canonical issue's reported Gateway Relay path. (apps/ios/Sources/Voice/TalkModeGatewayConfig.swift:277, a004e18b4bbd)
  • PR adds native WebRTC routing: PR head adds realtimeWebRTC route state and maps parsed OpenAI WebRTC configs to that native route while preserving explicit relay cases. (apps/ios/Sources/Voice/TalkModeGatewayConfig.swift:225, 9486162f26b4)
  • PR uses resolved route at runtime: PR head branches iOS Talk startup between native WebRTC and Gateway Relay using the resolved execution mode. (apps/ios/Sources/Voice/TalkModeManager.swift:471, 9486162f26b4)
  • Gateway client-owned WebRTC contract checked: The Gateway talk.client.create path is client-owned, rejects gateway-relay, and returns provider browser sessions when WebRTC is supported. (src/gateway/server-methods/talk-client.ts:94, 9486162f26b4)
  • Physical proof inspected: Downloaded still frames match the PR body hashes and show a physical iPhone in Ready and Listening states on Native WebRTC; the transcript records a completed prompt and assistant response.

Likely related people:

  • steipete: Authored the merged iOS Gateway speech-provider route split in fix(ios): use Gateway speech providers in Talk #98376 and authored the maintainer repair commits on this PR branch. (role: recent adjacent owner and repair author; confidence: high; commits: d0f655811cdc, c45319d308b9, 9486162f26b4; files: apps/ios/Sources/Voice/TalkModeGatewayConfig.swift, apps/ios/Sources/Voice/TalkModeManager.swift, apps/ios/Tests/TalkModeConfigParsingTests.swift)
  • ngutman: Authored the merged iOS realtime Talk mode work in [codex] improve iOS realtime talk mode #86355 that defines much of the surrounding WebRTC and Talk manager behavior. (role: recent iOS Talk realtime contributor; confidence: high; commits: 9ca52ce3d930; files: apps/ios/Sources/Voice/TalkModeManager.swift, apps/ios/Sources/Voice/TalkModeGatewayConfig.swift, apps/ios/Sources/Voice/TalkRealtimeWebRTCSession.swift)
  • Solvely-Colin: Authored merged iOS pro/gateway flow work in Refresh iOS pro UI and gateway flows #87367 that touched the iOS relay/config/manager surfaces this PR preserves for explicit relay behavior. (role: feature introducer and adjacent iOS relay contributor; confidence: medium; commits: f6e51ff99af4; files: apps/ios/Sources/Voice/RealtimeTalkRelaySession.swift, apps/ios/Sources/Voice/TalkModeGatewayConfig.swift, apps/ios/Sources/Voice/TalkModeManager.swift)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jul 1, 2026
@PollyBot13

PollyBot13 commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Proof media/link comment removed after merge for privacy. Validation evidence was summarized in the sanitized PR body.

1 similar comment
@PollyBot13

PollyBot13 commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Proof media/link comment removed after merge for privacy. Validation evidence was summarized in the sanitized PR body.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 1, 2026
@PollyBot13
PollyBot13 force-pushed the fix-ios-talk-realtime-routing-reviewed branch 2 times, most recently from 8c183d4 to cb26f4a Compare July 2, 2026 05:31
@steipete steipete self-assigned this Jul 2, 2026
@steipete
steipete force-pushed the fix-ios-talk-realtime-routing-reviewed branch from cb26f4a to 6c199bb Compare July 2, 2026 12:02
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jul 2, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 2, 2026
@openclaw-barnacle openclaw-barnacle Bot added app: android App: android size: L and removed size: M labels Jul 2, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 2, 2026
@steipete
steipete force-pushed the fix-ios-talk-realtime-routing-reviewed branch from fc19506 to 9486162 Compare July 2, 2026 19:27
@openclaw-barnacle openclaw-barnacle Bot removed channel: telegram Channel integration: telegram app: android App: android app: macos App: macos scripts Repository scripts commands Command implementations agents Agent runtime and tooling extensions: google labels Jul 2, 2026
@steipete
steipete force-pushed the fix-ios-talk-realtime-routing-reviewed branch from 9486162 to f20ed11 Compare July 2, 2026 19:38
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 2, 2026
@steipete
steipete merged commit 8b99c45 into openclaw:main Jul 2, 2026
114 of 116 checks passed
@steipete

steipete commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

@github-actions github-actions Bot mentioned this pull request Jul 2, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@steipete steipete

Labels

app: ios App: ios docs Improvements or additions to documentation gateway Gateway runtime merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: 🎥 video Contributor real behavior proof includes video or recording evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XL status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@PollyBot13 @steipete