Skip to content

feat(ios): add talk tab realtime playback#88105

Merged
ngutman merged 1 commit into
mainfrom
feat/ios-talk-tab-realtime-playback
May 29, 2026
Merged

feat(ios): add talk tab realtime playback#88105
ngutman merged 1 commit into
mainfrom
feat/ios-talk-tab-realtime-playback

Conversation

@ngutman
Copy link
Copy Markdown
Member

@ngutman ngutman commented May 29, 2026

Summary

  • Add a dedicated iOS Talk tab with realtime status, animated voice UI, voice mode details, and controls.
  • Make iOS realtime Talk playback response-scoped using additive gateway audioDone lifecycle events.
  • Suppress built-in speaker echo while preserving headset/external-route barge-in, and use realtime voice audio session routing.

Verification

  • node scripts/run-vitest.mjs src/gateway/server-methods/talk.test.ts extensions/openai/realtime-voice-provider.test.ts --reporter=dot
  • iOS device build for Nimrod’s iPhone via xcodebuild -project apps/ios/OpenClaw.xcodeproj -scheme OpenClaw -destination 'id=00008140-000848A92EE3001C' -configuration Debug build
  • Installed and launched on device with xcrun devicectl device install app and xcrun devicectl device process launch
  • Codex review clean: CODEX_REVIEW_AUTO_TESTS=0 ~/.pi/agent/skills/codex-review/scripts/codex-review --mode local --parallel-tests "node scripts/run-vitest.mjs src/gateway/server-methods/talk.test.ts extensions/openai/realtime-voice-provider.test.ts --reporter=dot"

Real behavior proof

Behavior addressed: iOS Talk now has a real tab and avoids self-listening loops by finishing response-scoped playback before re-enabling built-in speaker mic forwarding.
Real environment tested: Physical iPhone Nimrod’s iPhone using bundle ai.openclaw.ios.test.guti-gzs353x62e against the local OpenClaw gateway path.
Exact steps or command run after this patch: Built with xcodebuild for the connected iPhone, installed with devicectl, launched the app, and exercised the Talk flow during development with phone diagnostics.
Evidence after fix: Phone diagnostics showed realtime speaker echo suppression during output; focused gateway/provider tests passed; Codex review reported no accepted/actionable findings.
Observed result after fix: Talk reply playback completed without the previous repeated self-trigger loop in the tested flow.
What was not tested: App Store/TestFlight packaging, alternate Bluetooth/headset routes beyond code-path preservation, and broad full-suite CI.

@openclaw-barnacle openclaw-barnacle Bot added app: ios App: ios gateway Gateway runtime extensions: openai size: L maintainer Maintainer-authored PR labels May 29, 2026
@clawsweeper
Copy link
Copy Markdown
Contributor

clawsweeper Bot commented May 29, 2026 •
edited
Loading

Codex review: needs maintainer review before merge. Reviewed May 29, 2026, 3:50 PM ET / 19:50 UTC.

Summary
The PR adds a dedicated iOS Talk tab and makes realtime relay playback response-scoped by adding gateway audioDone events, OpenAI event IDs, iOS pending-output playback handling, and realtime audio-session routing.

PR surface: Source +52, Other +785. Total +837 across 8 files.

Reproducibility: not applicable. as a feature PR with a behavioral relay fix mixed in; I did not run the app or tests in this read-only review, but the PR discussion gives physical-device after-fix proof and the source path is clear.

Review metrics: 2 noteworthy metrics.

  • Relay lifecycle surface: 1 event type added, 2 optional ids propagated. The new audioDone, itemId, and responseId fields define the response boundary that iOS uses to finish playback without dropping follow-up audio.
  • iOS initial-tab aliases: 2 aliases added. The existing --openclaw-initial-tab launch surface now accepts talk and voice, which affects debug and automated iOS launch routing.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Confirm current required checks and decide whether the physical-device speaker-route proof is enough without Bluetooth/headset coverage.

Risk before merge

  • [P1] The new audioDone lifecycle is message-delivery-sensitive: a wrong response/item boundary could drop rapid follow-up assistant audio or keep built-in speaker microphone forwarding suppressed too long.
  • [P1] The PR body explicitly leaves App Store/TestFlight packaging, broad full-suite CI, and alternate Bluetooth/headset route proof out of scope, so maintainers still need to decide whether the focused physical-device proof is enough.

Maintainer options:

  1. Land after audio-boundary review (recommended)
    Maintain the additive audioDone design, but merge only after maintainers are satisfied that rapid follow-up audio and built-in speaker suppression behave as described on the latest head.
  2. Ask for broader route proof
    If maintainers want less residual risk, request headset/Bluetooth or broader CI proof before merge rather than changing the current implementation blindly.

Next step before merge

  • [P2] The remaining action is maintainer review of the response-scoped audio delivery behavior, route proof, and CI state, not a narrow automated repair.

Security
Cleared: No concrete security or supply-chain concern was found; the diff does not change dependencies, CI, package scripts, credentials, permissions, or downloaded code paths.

Review details

Best possible solution:

Land the additive iOS Talk tab and response-scoped relay lifecycle after maintainer review confirms the audio boundary behavior and current checks are acceptable, keeping the new gateway event ignore-compatible for older clients.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a feature PR with a behavioral relay fix mixed in; I did not run the app or tests in this read-only review, but the PR discussion gives physical-device after-fix proof and the source path is clear.

Is this the best way to solve the issue?

Yes, with maintainer review: the additive gateway event plus iOS response-scoped playback queue is the narrow maintainable shape for this PR's goal, while broader route coverage can stay as proof/merge judgment rather than a code rewrite.

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 1e2fda9e6886.

Label changes

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body and latest comments report after-fix proof on a physical iPhone with build, install, launch, Talk flow exercise, diagnostics, and focused tests on the latest head.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: This is a normal-priority iOS Talk and realtime relay improvement with limited blast radius but meaningful audio-delivery review needs.
  • merge-risk: 🚨 message-delivery: The PR changes assistant audio lifecycle boundaries and iOS pending-output playback, so a subtle ordering bug could drop or delay realtime audio.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body and latest comments report after-fix proof on a physical iPhone with build, install, launch, Talk flow exercise, diagnostics, and focused tests on the latest head.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body and latest comments report after-fix proof on a physical iPhone with build, install, launch, Talk flow exercise, diagnostics, and focused tests on the latest head.
Evidence reviewed

PR surface:

Source +52, Other +785. Total +837 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 3 53 1 +52
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 5 802 17 +785
Total 8 855 18 +837

What I checked:

Likely related people:

  • Solvely-Colin: Recent path history shows the iOS pro UI and realtime relay mode were introduced or heavily expanded in commits affecting RootTabs, RealtimeTalkRelaySession, and gateway relay code. (role: feature introducer; confidence: high; commits: f6e51ff99af4, e730e9bd0b83; files: apps/ios/Sources/Voice/RealtimeTalkRelaySession.swift, apps/ios/Sources/RootTabs.swift, src/gateway/talk-realtime-relay.ts)
  • steipete: Recent history and local blame point to repeated Talk realtime relay, forced-consult, OpenAI realtime, and iOS gateway maintenance across the affected surfaces. (role: recent area contributor; confidence: high; commits: bbf9c45ba7a8, 5dccba7405c4, a1f47bccb59a; files: src/gateway/talk-realtime-relay.ts, apps/ios/Sources/Voice/TalkModeManager.swift, extensions/openai/realtime-voice-provider.ts)
  • vincentkoc: Recent OpenAI voice-provider and Talk diagnostics commits touch the provider/catalog and observability side of the realtime voice path used by this PR. (role: adjacent provider and diagnostics contributor; confidence: medium; commits: 27b15a19e84c, 16321a27b64c; files: extensions/openai/realtime-voice-provider.ts, src/talk/session-log-runtime.ts)
  • ngutman: Although also the current PR author, prior merged history shows iOS gateway recovery and gateway UX work on adjacent app surfaces, making this a plausible routing candidate beyond authorship alone. (role: recent iOS gateway contributor; confidence: medium; commits: 00a0858fd9dc, 6380c872bcc2; files: apps/ios/Sources/RootTabs.swift, apps/ios/Sources/Gateway/GatewayConnectionController.swift)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 29, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b59cd9971b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/ios/Sources/Voice/RealtimeTalkRelaySession.swift
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels May 29, 2026
@ngutman ngutman force-pushed the feat/ios-talk-tab-realtime-playback branch from b59cd99 to 104df8c Compare May 29, 2026 19:17
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels May 29, 2026
@ngutman ngutman force-pushed the feat/ios-talk-tab-realtime-playback branch from 104df8c to eb910d1 Compare May 29, 2026 19:29
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels May 29, 2026
@ngutman ngutman force-pushed the feat/ios-talk-tab-realtime-playback branch from eb910d1 to 4d0b3e9 Compare May 29, 2026 19:41
@ngutman
Copy link
Copy Markdown
Member Author

ngutman commented May 29, 2026

Addressed the review follow-ups on the latest head (4d0b3e9915):

  • Added Sources/Design/TalkProTab.swift to apps/ios/SwiftSources.input.xcfilelist so the new Talk tab is covered by the iOS SwiftFormat/SwiftLint prebuild filelist.
  • Ran SwiftFormat on TalkProTab.swift and wrapped the realtime suppression log line so SwiftLint stays clean.
  • Resolved the stale Codex playback-thread comment after the latest code queues follow-up audio during PCM drain and replays it into a fresh stream.

Verification:

  • git diff --check
  • cd apps/ios && swiftformat --lint --config ../../config/swiftformat --unexclude Sources --filelist SwiftSources.input.xcfilelist
  • cd apps/ios && SCRIPT_INPUT_FILE_LIST_COUNT=1 SCRIPT_INPUT_FILE_LIST_0=SwiftSources.input.xcfilelist swiftlint lint --config ../../config/swiftlint.yml --use-script-input-file-lists
  • node scripts/run-vitest.mjs src/gateway/server-methods/talk.test.ts extensions/openai/realtime-voice-provider.test.ts --reporter=dot
  • pnpm tsgo:core
  • cd apps/ios && xcodegen generate
  • xcodebuild -project apps/ios/OpenClaw.xcodeproj -scheme OpenClaw -destination 'id=6A19D82B-4EA6-5D67-B3A7-0AB3B71B550C' -configuration Debug -allowProvisioningUpdates build

@clawsweeper re-review

@clawsweeper
Copy link
Copy Markdown
Contributor

clawsweeper Bot commented May 29, 2026 •
edited
Loading

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@ngutman
Copy link
Copy Markdown
Member Author

ngutman commented May 29, 2026

Latest head was rebuilt, installed, and launched on Nimrod’s iPhone after the filelist/format follow-up.

Device proof:

  • xcodebuild -project apps/ios/OpenClaw.xcodeproj -scheme OpenClaw -destination 'id=6A19D82B-4EA6-5D67-B3A7-0AB3B71B550C' -configuration Debug -allowProvisioningUpdates build
  • xcrun devicectl device install app --device 6A19D82B-4EA6-5D67-B3A7-0AB3B71B550C .../OpenClaw.app
  • xcrun devicectl device process launch --device 6A19D82B-4EA6-5D67-B3A7-0AB3B71B550C ai.openclaw.ios.test.guti-gzs353x62e

@clawsweeper re-review

@clawsweeper
Copy link
Copy Markdown
Contributor

clawsweeper Bot commented May 29, 2026 •
edited
Loading

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels May 29, 2026
@github-actions github-actions Bot added the dependencies-changed PR changes dependency-related files label May 29, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 29, 2026 •
edited
Loading

Dependency graph guard cleared

This PR no longer has blocked dependency graph changes. A future dependency graph change requires a fresh /allow-dependencies-change comment after the guard blocks that new head SHA.

  • Current SHA: f41112a882a3fc2776350e99aeb92c166fb93bc3

@ngutman ngutman force-pushed the feat/ios-talk-tab-realtime-playback branch from d5f6af3 to 4d0b3e9 Compare May 29, 2026 19:58
@github-actions github-actions Bot removed the dependencies-changed PR changes dependency-related files label May 29, 2026
@ngutman
Copy link
Copy Markdown
Member Author

ngutman commented May 29, 2026

Merged via squash.

Thanks @ngutman!

@github-actions github-actions Bot mentioned this pull request May 29, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

No one assigned

Labels

app: ios App: ios extensions: openai gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ngutman