Skip to content

fix(ios): keep gateway speech providers native#98212

Closed
lin-hongkuan wants to merge 4 commits into
openclaw:mainfrom
lin-hongkuan:lhk/ios-talk-gateway-tts-98153
Closed

fix(ios): keep gateway speech providers native#98212
lin-hongkuan wants to merge 4 commits into
openclaw:mainfrom
lin-hongkuan:lhk/ios-talk-gateway-tts-98153

Conversation

@lin-hongkuan

@lin-hongkuan lin-hongkuan commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • keep iOS Gateway Default speech providers such as Xiaomi/Microsoft on the native Talk path instead of treating every non-ElevenLabs provider as realtime
  • route non-local native speech providers through the Gateway talk.speak RPC before falling back to iOS system speech
  • add iOS Talk regression coverage for speech-only Gateway providers and local/realtime provider routing guards

Issue / Motivation

Fixes #98153.

Current iOS Talk enables realtime whenever the resolved Gateway provider is not ElevenLabs, so a speech/TTS provider with no realtime config starts the realtime path and then falls back to iOS system speech instead of using the configured Gateway TTS provider.

What Problem This Solves

The iOS app now distinguishes Gateway realtime mode from Gateway speech/TTS providers. Speech-only providers stay on the native Talk path and call the Gateway talk.speak RPC, so configuring a non-ElevenLabs Gateway TTS provider no longer silently falls back to system speech.

Changes

  • Derive iOS realtime mode from the parsed execution mode rather than from the active speech provider id.
  • Add a native Gateway talk.speak playback path for non-local speech providers, including directive params and PCM-vs-compressed playback selection from outputFormat.
  • Keep ElevenLabs/system/OpenAI realtime routing out of the Gateway-speak fallback helper.
  • Add DEBUG test hooks and Swift Testing regressions for the Gateway speech provider path.

Testing

  • node scripts/run-vitest.mjs src/gateway/server-methods/talk.test.ts — passed locally: 62 tests passed across gateway client/method shards.
  • git diff --check — passed.
  • Added-line secret scan over git diff origin/main...HEAD — SECRET_SCAN_MATCHES 0.
  • pnpm ios:gen — attempted, blocked on this Linux host because xcodegen is not installed after signing/version prep.
  • pnpm exec tsc --noEmit --pretty false — attempted, Node heap OOM at default limit.
  • NODE_OPTIONS=--max-old-space-size=4096 pnpm exec tsc --noEmit --pretty false — attempted, timed out after 600s.
  • xcodebuild could not be run in this environment because the host does not provide Xcode/iOS Simulator tooling.

Evidence

  • The Gateway talk.speak regression test covers the server RPC contract used by the new iOS native speech path and passed locally with 62 tests.
  • The iOS patch adds Swift Testing coverage for:
    • non-local Gateway speech providers staying in native mode instead of realtime;
    • Gateway speech directives preserving provider/voice/model/speed/output format;
    • local/system providers and realtime providers being excluded from the Gateway-speak fallback helper.
  • Local verification on 2026-07-01:
    • src/gateway/server-methods/talk.test.ts: 62 tests passed.
    • git diff --check: clean.
    • added-line secret scan: SECRET_SCAN_MATCHES 0.
  • Live iOS build/simulator proof is not claimed here; it requires macOS/Xcode tooling unavailable on this Linux runner.

@openclaw-barnacle openclaw-barnacle Bot added app: ios App: ios size: M triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 30, 2026
@clawsweeper

clawsweeper Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 30, 2026, 9:19 PM ET / 01:19 UTC.

Summary
This PR changes iOS Talk routing so Gateway speech providers stay native, adds an iOS Gateway talk.speak playback helper, and adds Swift regression coverage for speech-provider routing and readiness.

PR surface: Other +265. Total +265 across 2 files.

Reproducibility: yes. at source level. Current main still misclassifies non-ElevenLabs Gateway speech providers as realtime and lacks iOS Gateway talk.speak playback; I did not run a live iOS device or simulator repro.

Review metrics: 1 noteworthy metric.

  • Native TTS paths: 1 of 2 routed through Gateway talk.speak. The diff adds Gateway speech to final playback while the always-enabled incremental path still falls back through ElevenLabs/system, which affects merge readiness.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #98153
Summary: This PR is a candidate fix for the canonical iOS Gateway Default speech-provider misclassification and missing talk.speak playback issue, but it is not merge-ready because incremental playback and real behavior proof remain unresolved.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦪 silver shellfish
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Route incremental speech segments through the same Gateway talk.speak path, or disable incremental speech for Gateway speech providers so final playback handles them.
  • [P1] Add redacted iOS simulator or device proof showing Gateway Default with Xiaomi or Microsoft stays Native and plays Gateway audio; redact IP addresses, API keys, phone numbers, endpoints, and other private details.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body lists tests, CI/build status, and local checks but explicitly does not include redacted iOS simulator/device output, logs, recording, or linked artifacts showing the fixed Gateway talk.speak behavior; after adding proof, update the PR body to trigger a fresh ClawSweeper review or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

  • [P1] Existing Gateway Default users may still hear iOS system voice for streamed assistant replies because the always-enabled incremental speech path remains ElevenLabs/system-only.
  • [P2] The PR changes provider routing and fallback behavior for existing iOS Talk setups without redacted simulator/device proof that a non-ElevenLabs Gateway provider stays native and plays Gateway audio.
  • [P1] Another open PR targets the same canonical bug, so maintainers need one final branch to avoid overlapping iOS Talk implementations.

Maintainer options:

  1. Fix incremental Gateway playback before merge (recommended)
    Extend the Gateway talk.speak path to incremental speech segments, or disable incremental speech for Gateway speech providers so final playback handles them, then add redacted iOS runtime proof.
  2. Choose one canonical iOS branch
    Compare this PR with fix(ios/talk): use Gateway talk.speak for non-ElevenLabs speech providers #98214 and continue only with the branch that cleanly covers final and incremental playback plus proof.
  3. Accept reduced streaming behavior explicitly
    Maintainers could intentionally merge with streamed replies still falling back to system voice, but that should be an explicit compatibility decision because it leaves part of the reported workflow broken.

Next step before merge

  • [P1] Manual review remains necessary because the branch needs a code repair plus contributor or maintainer iOS runtime proof; automation should not queue a repair marker while proof is absent.

Security
Cleared: The diff touches iOS Gateway RPC playback logic and Swift tests without new dependencies, workflows, lockfiles, package execution paths, or secret-handling expansion.

Review findings

  • [P1] Route incremental speech through Gateway talk.speak — apps/ios/Sources/Voice/TalkModeManager.swift:2329-2333
Review details

Best possible solution:

Land one canonical iOS fix that routes both final and incremental native speech through Gateway talk.speak for speech providers, preserves explicit ElevenLabs/OpenAI behavior, passes iOS CI, and includes redacted iOS simulator or device proof.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main still misclassifies non-ElevenLabs Gateway speech providers as realtime and lacks iOS Gateway talk.speak playback; I did not run a live iOS device or simulator repro.

Is this the best way to solve the issue?

No, not yet. The iOS routing/playback boundary is the right fix location, but the branch must also cover the always-enabled incremental speech path and provide real iOS proof.

Full review comments:

  • [P1] Route incremental speech through Gateway talk.speak — apps/ios/Sources/Voice/TalkModeManager.swift:2329-2333
    The PR adds Gateway talk.speak only to playAssistant, but iOS enables incremental TTS for normal streamed replies and speakIncrementalSegment still requires ElevenLabs, falling back to system speech otherwise. That means Xiaomi/Microsoft Gateway Default replies can still skip the configured Gateway TTS provider whenever assistant text streams in segments; please route incremental segments through talk.speak or disable incremental TTS for Gateway speech providers so the final Gateway path handles them.
    Confidence: 0.93

Overall correctness: patch is incorrect
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3669280b9aec.

Label changes

Label justifications:

  • P1: This PR targets a broken user-facing iOS Talk workflow where configured Gateway speech providers are ignored, and the current head still has a central playback-path blocker.
  • merge-risk: 🚨 compatibility: The diff changes iOS Talk routing and fallback order for existing Gateway Default, ElevenLabs, system speech, and explicit realtime setups.
  • merge-risk: 🚨 auth-provider: The core behavior changes provider and credential ownership for Gateway-managed speech providers versus local ElevenLabs and OpenAI realtime routing.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists tests, CI/build status, and local checks but explicitly does not include redacted iOS simulator/device output, logs, recording, or linked artifacts showing the fixed Gateway talk.speak behavior; after adding proof, update the PR body to trigger a fresh ClawSweeper review or ask a maintainer to comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Other +265. Total +265 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 2 272 7 +265
Total 2 272 7 +265

What I checked:

  • Repository policy applied: Root AGENTS.md was read fully; no scoped AGENTS.md exists under apps/ios, and the provider-routing compatibility and real-behavior-proof rules apply to this PR. (AGENTS.md:1, 3669280b9aec)
  • Current main still has the reported routing bug: Current main still derives realtime mode from activeProvider != Self.defaultTalkProvider || executionMode != .native, which source-reproduces the Gateway speech provider misclassification described by the linked issue. (apps/ios/Sources/Voice/TalkModeManager.swift:2680, 3669280b9aec)
  • PR head adds only final-playback Gateway speak: The PR head adds canUseGatewayTalkSpeak handling in playAssistant, so final fallback playback can call Gateway talk.speak. (apps/ios/Sources/Voice/TalkModeManager.swift:1547, 5cfc9e99fa33)
  • PR head leaves incremental speech ElevenLabs/system-only: The PR head still enables incremental TTS unconditionally, and speakIncrementalSegment falls back to TalkSystemSpeechSynthesizer whenever the context cannot use ElevenLabs, so streamed replies for Xiaomi/Microsoft can still skip Gateway talk.speak. (apps/ios/Sources/Voice/TalkModeManager.swift:2329, 5cfc9e99fa33)
  • Tests cover readiness but not incremental Gateway playback: The added Swift tests cover native mode, Gateway-owned readiness, and provider guard behavior, but they do not exercise incremental speech segments using Gateway talk.speak. (apps/ios/Tests/TalkModeConfigParsingTests.swift:313, 5cfc9e99fa33)
  • Native Talk contract: The Talk docs define native iOS/macOS/Android Talk as local speech recognition, Gateway chat, and talk.speak TTS, which supports the intended fix direction and the missing incremental coverage concern. Public docs: docs/nodes/talk.md. (docs/nodes/talk.md:11, 3669280b9aec)

Likely related people:

  • ngutman: Recent merged history touched iOS Talk realtime fallback, voice-mode display, playback, and parsing tests around the same manager boundary. (role: recent iOS Talk contributor; confidence: high; commits: 47dbc675e953, 6897711d1991, 19e4c37c3775; files: apps/ios/Sources/Voice/TalkModeManager.swift, apps/ios/Tests/TalkModeConfigParsingTests.swift)
  • Solvely-Colin: Path history shows iOS gateway and realtime Talk flow work that wired the same TalkModeManager routing area. (role: iOS gateway flow contributor; confidence: medium; commits: f6e51ff99af4, e730e9bd0b83; files: apps/ios/Sources/Voice/TalkModeManager.swift, apps/ios/Sources/Voice/TalkModeGatewayConfig.swift, apps/ios/Tests/TalkModeConfigParsingTests.swift)
  • mbelinky: History shows prior iOS incremental speech and TTS fallback work, which is directly relevant to the remaining incremental Gateway playback gap. (role: incremental iOS speech contributor; confidence: medium; commits: a36ccf4156f3, bf7061092a1e; files: apps/ios/Sources/Voice/TalkModeManager.swift)
  • steipete: History includes shared Apple Talk config parsing, native node Talk handoff, and sibling native talk.speak behavior used to judge this fix boundary. (role: adjacent native Talk contributor; confidence: medium; commits: 4f482d2a2b4d, 466f7183207d, 16a5f0b00691; files: apps/ios/Sources/Voice/TalkModeManager.swift, apps/ios/Tests/TalkModeConfigParsingTests.swift, apps/macos/Sources/OpenClaw/TalkModeRuntime.swift)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jun 30, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 30, 2026
Closed
7 tasks
@lin-hongkuan

lin-hongkuan commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Updated the PR body with the required ## What Problem This Solves and ## Evidence sections and pushed commit 6e5014a7e2 to wrap the native Gateway provider guard for SwiftFormat. Local validation after the code update: node scripts/run-vitest.mjs src/gateway/server-methods/talk.test.ts (62 passed), git diff --check, and added-line secret scan.

@clawsweeper

clawsweeper Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@lin-hongkuan

lin-hongkuan commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Updated the PR body to include the required ## What Problem This Solves and ## Evidence headings with fresh local validation from 2026-07-01.

@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 30, 2026
@lin-hongkuan

lin-hongkuan commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a follow-up fix for the red iOS compile blocker and the gateway-owned native speech provider readiness gap:

  • avoids concatenating OSLogMessage fragments in the talk.speak log line
  • treats native gateway talk.speak providers (for example Xiaomi/Microsoft) as gateway-owned credentials so they stay .ready instead of requiring a local provider API key
  • adds regression coverage for gateway speech providers remaining native/ready without local credentials

Validation on this Linux runner:

  • pnpm ios:build — blocked before compile because this runner has no Xcode/iOS tooling (xcodegen: command not found)
  • git diff --check — pass
  • diff secret scan for GitHub/OpenAI/Google/AWS/env key patterns — 0 hits

CI should now be able to exercise the macOS ios-build and Periphery jobs on the pushed head. I do not have simulator/device live audio proof from this environment.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot mentioned this pull request Jul 1, 2026
Merged
@github-actions github-actions Bot added duplicate This issue or pull request already exists close:duplicate Closed as duplicate dedupe:child Duplicate issue/PR child in dedupe cluster labels Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the fix. This is now covered by the landed #98376 / commit d0f6558.

Evidence: shared issue(s): #98153; overlapping changed hunks; shared file(s): apps/ios/Sources/Voice/TalkModeManager.swift.

Closing #98212 as a duplicate.

@github-actions github-actions Bot closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

app: ios App: ios close:duplicate Closed as duplicate dedupe:child Duplicate issue/PR child in dedupe cluster duplicate This issue or pull request already exists merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug(ios/talk): Gateway speech TTS provider is treated as realtime and falls back to iOS system voice

1 participant

@lin-hongkuan