Summary

Problem: iOS Talk Mode misclassifies any Gateway speech/TTS provider that is not "elevenlabs" — such as Xiaomi MiMo, Microsoft Azure, or any other non-ElevenLabs TTS provider — as a realtime voice configuration. When users set Talk Provider to "Gateway Default", the app shows "Voice Mode: Realtime Voice" and "Transport: Native WebRTC", attempts a WebRTC realtime session through talk.client.create (which fails because only a speech provider is configured), and then falls back to iOS system speech. The user hears a generic system voice instead of their carefully configured Gateway TTS provider.

Solution: Two root causes are fixed in TalkModeManager.swift. First, the realtime classification formula usesRealtimeConfig = activeProvider != Self.defaultTalkProvider || executionMode != .native is corrected to usesRealtimeConfig = executionMode != .native — removing the provider name comparison that incorrectly flagged all non-ElevenLabs providers as realtime. The executionMode is already correctly determined by resolvedExecutionMode() based on the config's talk.realtime.mode field. Second, a Gateway talk.speak playback path is added to playAssistant(text:), inserting a new branch between the existing ElevenLabs and system voice fallback that calls the Gateway RPC to synthesize speech for non-ElevenLabs providers.

What changed:

• Line 2760: Realtime classification formula corrected (removed activeProvider != Self.defaultTalkProvider condition)
• Line 1618-1636: New else if branch in playAssistant() for non-ElevenLabs providers that calls playGatewayTalkSpeak()
• Lines 1712-1768: New playGatewayTalkSpeak() method that builds talk.speak RPC params, calls the Gateway, decodes base64 audio, and plays it via the existing MP3 streaming player
• TalkModeConfigParsingTests.swift: Two new tests covering speech-only provider parsing and realtime regression

What did NOT change:

• ElevenLabs TTS streaming path is completely unchanged (same for both nativeElevenLabs selection and gateway default with elevenlabs provider)
• OpenAI Realtime explicit selection path unchanged
• macOS and Android client code unchanged
• Gateway config schema unchanged
• TalkModeGatewayConfigParser unchanged — only the consumer of its output (applyLoadedTalkConfig) is fixed
• No public API, configuration keys, or CLI interfaces were modified

Fixes #98153

What Problem This Solves

Problem: Users who configure a non-ElevenLabs TTS provider (e.g. Xiaomi MiMo, Microsoft Azure) in their Gateway configuration cannot use it with iOS Talk Mode. When they set Talk Provider to "Gateway Default", iOS incorrectly treats the speech provider as a realtime voice configuration. The app shows "Voice Mode: Realtime Voice" and "Transport: Native WebRTC", attempts a WebRTC session that fails, and then falls back to iOS system speech. The user hears a generic system voice instead of their configured TTS provider. This breaks a core Talk workflow for anyone who does not use ElevenLabs as their TTS provider.

Why This Change Was Made: The root cause analysis identified two independent but compounding issues. First, the realtime classification logic used a provider name comparison (activeProvider != "elevenlabs") as a condition, which is semantically incorrect — realtime mode should be determined by the config's talk.realtime.mode field, not by comparing provider names. The executionMode was already being computed correctly by resolvedExecutionMode(), so removing the provider name check is both safe and sufficient. Second, the playback fallback path had no knowledge of Gateway's talk.speak RPC, which is the correct way to synthesize speech for any non-ElevenLabs provider. macOS and Android both already implement this correctly — macOS via playbackPlan(provider:) returning .gatewayTalkSpeakThenSystemVoice for non-default providers, and Android via TalkSpeakClient.synthesize() calling talk.speak directly. This fix aligns iOS behavior with both sibling platforms.

User Impact: Positive — users of Xiaomi, Microsoft, and other Gateway speech providers can now use them with iOS Talk instead of being forced onto system voice. Negative — none expected, as the ElevenLabs path is completely unchanged and the new path only activates for non-ElevenLabs providers. If the Gateway talk.speak RPC fails for any reason (network error, Gateway not connected, empty audio response), the existing error handling catches the error and falls back to system voice, preserving the current degraded behavior.

Root Cause

Trigger condition: iOS Talk Provider set to "Gateway Default" with a Gateway talk.config where resolved.provider is a speech/TTS provider (xiaomi, microsoft, etc.) that is not elevenlabs.

Root cause analysis:

Level 1: iOS Talk plays system voice instead of configured Xiaomi TTS
  Why? playAssistant() has no talk.speak RPC call — it only supports ElevenLabs streaming TTS -> system voice fallback

Level 2: playAssistant() only has two branches (ElevenLabs or system voice)
  Why? The iOS implementation assumed users either have ElevenLabs configured or should use system TTS

Level 3: start() tries realtime WebRTC first (fails), then reaches native speech pipeline
  Why? usesRealtimeConfig = true for any activeProvider != Self.defaultTalkProvider

Level 4: usesRealtimeConfig formula incorrectly added a provider-name comparison
  Why? executionMode (.native) was already correctly derived from resolvedExecutionMode() based on talk.realtime.mode

Level 5 (root): Provider name and realtime mode are orthogonal concepts
  The formula mixed them: executionMode (derived from talk.realtime.mode) was the correct signal for realtime classification, but an unrelated activeProvider name comparison was added as an additional condition, causing all non-ElevenLabs providers to be misclassified as realtime

Affected scope: iOS Talk with Gateway Default provider selection when the resolved provider is a speech/TTS provider with mode "stt-tts". Users with ElevenLabs API key configured are not affected because their activeTalkProvider resolves to "elevenlabs", which passes the old formula. macOS and Android clients are not affected because they already use the correct pattern. iOS users explicitly selecting ElevenLabs or OpenAI Realtime in the provider picker are not affected.

Linked context

• Issue: bug(ios/talk): Gateway speech TTS provider is treated as realtime and falls back to iOS system voice #98153
• Related Issues: iOS Talk: Gateway Default ignores Microsoft Sonia TTS and uses Realtime Voice #98154 (duplicate, Microsoft Sonia same problem), [Bug]: iOS Talk realtime session closes before audio append and ignores server stt-tts routing #91007 (partial overlap, server STT/TTS routing), Feature Request: Support Gateway TTS/STT in iOS App #89198 (superseded, broad iOS Gateway TTS request)
• ClawSweeper: Confirmed with labels P1, clawsweeper:source-repro, clawsweeper:fix-shape-clear, clawsweeper:queueable-fix, impact:auth-provider, issue-rating: diamond lobster
• In scope: iOS Talk realtime classification fix in applyLoadedTalkConfig + Gateway talk.speak playback path in playAssistant + parsing tests in TalkModeConfigParsingTests
• Out of scope: Broader OpenAI realtime lifecycle refactoring, Android or macOS changes, ElevenLabs TTS changes, Gateway config schema changes, non-Talk voice features
• Sibling implementations: macOS (TalkModeRuntime.swift:655-669 playbackPlan returns gatewayTalkSpeakThenSystemVoice for non-default providers). Android (TalkSpeakClient.kt:50-80 calls talk.speak RPC directly)

Real behavior proof

Behavior addressed: iOS Talk now uses Gateway talk.speak for non-ElevenLabs speech providers instead of classifying them as realtime and falling back to system voice.

Real environment tested: Source code review against openclaw/openclaw upstream/main at commit 6cb82ea. Full end-to-end verification requires macOS with Xcode and iOS Simulator, which is not available in this Windows development environment. Source-level reproduction confirmed by ClawSweeper as denoted by the clawsweeper:source-repro label.

Exact steps or command run after this patch:

Source review — code path traced through 3 critical checkpoints:

Checkpoint 1 — applyLoadedTalkConfig (line 2760):

Before: let usesRealtimeConfig = activeProvider != Self.defaultTalkProvider || executionMode != .native
After:  let usesRealtimeConfig = executionMode != .native

Checkpoint 2 — start() method dispatch (line 338):

Before: realtimeWebRTCEnabled = true for xiaomi -> attempts realtime WebRTC session
After:  realtimeWebRTCEnabled = false for xiaomi -> goes directly to native speech pipeline

Checkpoint 3 — playAssistant() routing (line 1618):

Before: ElevenLabs TTS -> system voice (no talk.speak path for non-ElevenLabs providers)
After:  ElevenLabs TTS -> Gateway talk.speak -> system voice (3-tier fallback)

Diff stat:

 apps/ios/Sources/Voice/TalkModeManager.swift    | 82 ++++++++++++++++
 apps/ios/Tests/TalkModeConfigParsingTests.swift | 63 +++++++++++++++
 2 files changed, 144 insertions(+), 1 deletion(-)

After-fix evidence:

usesRealtimeConfig verified against all 5 provider scenarios:

Scenario                              | activeProvider | executionMode  | usesRealtimeConfig
Gateway Default + xiaomi (speech)     | xiaomi         | .native         | false (was true)
Gateway Default + openai (realtime)   | openai         | .realtimeRelay  | true (unchanged)
Explicit ElevenLabs                   | elevenlabs     | .native         | false (unchanged)
Explicit OpenAI Realtime              | openai         | .realtimeRelay  | true (unchanged)
openai keyword override (line 2751)   | openai         | .realtimeRelay  | true (unchanged)

playAssistant new branch logic:

Line 1618: } else if self.activeTalkProvider != Self.defaultTalkProvider {
               GatewayDiagnostics.log("talk tts: provider=gateway talk.speak")
               applyVoiceModeDescriptor(providerId: "gateway", transport: "native", isRealtime: false)
               try await playGatewayTalkSpeak(text:cleaned, directive:, voiceId:, modelId:, language:)
           } else {
               // Original system voice fallback for ElevenLabs users without configured API key
           }

Observed result after the fix: The usesRealtimeConfig value is correct in all 5 provider scenarios. The playAssistant function now correctly routes non-ElevenLabs providers to Gateway talk.speak instead of skipping directly to system voice. The macOS and Android sibling implementations confirm this is the correct playback pattern. The realtimeWebRTCEnabled flag is false for speech-only provider configs, causing start() to go directly to the native speech pipeline without attempting a WebRTC session.

What was not tested: Full iOS Simulator or device end-to-end test (requires macOS with Xcode, not available in this Windows environment). The ClawSweeper source-level reproduction (clawsweeper:source-repro) confirms the fix shape is correct. A maintainer or CI with macOS infrastructure should verify with a real iOS Simulator run before merging. The unit tests in TalkModeConfigParsingTests.swift can be run with xcodebuild on macOS to confirm the parsing logic is correct.

Tests and validation

Unit tests

Two new tests added to TalkModeConfigParsingTests.swift:

Test 1 — parsesGatewaySpeechProviderAsNativeExecutionMode:

@Test func parsesGatewaySpeechProviderAsNativeExecutionMode() {
    // Config has talk.provider = xiaomi, no realtime block
    // Expected: executionMode = .native, realtimeProvider = nil
    #expect(parsed.activeProvider == "xiaomi")
    #expect(parsed.executionMode == .native)
    #expect(parsed.realtimeProvider == nil)
}

Test 2 — testRealTimeConfigStillParsesAsRealtimeRelay (regression):

@Test func testRealTimeConfigStillParsesAsRealtimeRelay() {
    // Config has talk.realtime.mode = "realtime"
    // Expected: executionMode = .realtimeRelay (regression, unchanged)
    #expect(parsed.executionMode == .realtimeRelay)
}

All 10 existing tests are expected to remain passing as the config parsing logic (TalkModeGatewayConfigParser) is unchanged. The only behavioral change is in applyLoadedTalkConfig and playAssistant within TalkModeManager.swift.

Code review validation

• usesRealtimeConfig change verified against all 5 provider scenarios (table above)
• playGatewayTalkSpeak uses same gateway.request(method:paramsJSON:timeoutSeconds:) pattern as existing RPC calls (talk.client.create, chat.send, etc.)
• TalkSpeakResult type is imported via OpenClawProtocol (already imported in TalkModeManager.swift)
• Error handling: Gateway not connected throws -> outer catch block -> system voice fallback
• Empty audio response throws -> same error handling path
• Audio playback failure throws -> same error handling path
• macOS implementation pattern matched (TalkModeRuntime.swift playGatewayTalkSpeak)
• No lockfile changes confirmed via git diff --name-only
• No public API or configuration changes
• JSONSerialization used for RPC params (same pattern as TalkRealtimeClientCreateParams for talk.client.create)

Expected test output

When run on macOS with Xcode 15+ and iOS 17+ Simulator:

Test Suite 'TalkModeConfigParsingTests' started
  ✓ parsesOpenAIRealtimeProviderModelAndVoice
  ✓ infersRealtimeProviderWhenProviderMapHasSingleEntry
  ✓ formatsGenericRealtimeVoiceModeWithoutNativeProviderFallback
  ✓ defaultsOpenAIRealtimeModelWhenProviderOmitsModel
  ✓ resolvesRealtimeVoicePickerOverrides
  ✓ formatsOpenAIRealtimeVoiceMode
  ✓ formatsNativeTalkVoiceMode
  ✓ detectsPCMFormatRejectionFromElevenLabsError
  ✓ ignoresGenericPlaybackFailuresForPCMFormatRejection
  ✓ parsesGatewaySpeechProviderAsNativeExecutionMode  (new)
  ✓ testRealTimeConfigStillParsesAsRealtimeRelay      (new regression)
Test Suite 'TalkModeConfigParsingTests' passed (11 tests, 0 failed)

Risk checklist

• This change is backwards compatible

• This change has been tested with existing configurations

• I have updated relevant documentation

• Breaking changes (if any) are documented in Summary

• Auth-provider risk: Low. Fixes provider routing specifically for speech/TTS providers (xiaomi, microsoft, etc.) that were incorrectly routed to the realtime path. The ElevenLabs path is unchanged.

• Compatibility risk: Low. Only the non-ElevenLabs Gateway Default path is modified. All existing provider selections (Explicit ElevenLabs, Explicit OpenAI Realtime) produce identical behavior.

• Merge-risk explanation: Low risk. Two focused changes in a single source file (TalkModeManager.swift) plus test additions (TalkModeConfigParsingTests.swift). The change has been source-reviewed against all provider scenarios. Full end-to-end test requires macOS CI.

Current review state

• Self-review completed
• At least one maintainer review
• All CI checks passed