feat(voice): universal talk waveform driven by real audio levels#102901
Conversation
openclaw-barnacle
Bot
added
docs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ea6268d90b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case .idle: .idle | ||
| case .thinking: .thinking | ||
| case .listening: .listening(level: self.level, speechActive: false) | ||
| case .speaking: .speaking(level: self.level) |
There was a problem hiding this comment.
When macOS speech uses paths without playback metering, such as playMP3 or playSystemVoice, the runtime only switches the phase to .speaking and never publishes a playback level, so TalkOverlayController.model.level remains 0. Passing that as .speaking(level: 0) disables the new TalkWaveformView synthetic fallback, causing MP3/system voice replies to render a flat low-power wave while audio is playing; carry an optional playback level through the overlay or pass nil for those unmetered speaking paths.
Useful? React with 👍 / 👎.
|
Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 11:03 AM ET / 15:03 UTC. Summary PR surface: Docs 0, Other +1231. Total +1231 across 38 files. Reproducibility: yes. source inspection gives a high-confidence reproduction path for the blocker: macOS unmetered MP3/system speech sets the phase to speaking, leaves the overlay level at 0, and the PR maps that to .speaking(level: 0) instead of nil. Review metrics: none identified. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Preserve the shared waveform direction, but carry optional playback-level state through the macOS overlay, resolve the merge conflicts, and add redacted real-device or simulator proof that the visible waveform follows live audio on the affected surfaces. Do we have a high-confidence way to reproduce the issue? Yes, source inspection gives a high-confidence reproduction path for the blocker: macOS unmetered MP3/system speech sets the phase to speaking, leaves the overlay level at 0, and the PR maps that to .speaking(level: 0) instead of nil. Is this the best way to solve the issue? No, not yet: the shared waveform approach is a maintainable direction, but the macOS overlay needs optional level semantics so unmetered paths keep the documented synthetic speaking fallback. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 36227737c547. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Docs 0, Other +1231. Total +1231 across 38 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
clawsweeper
Bot
added
rating: 🧂 unranked krab
One shared Siri-style talk animation across iOS, watchOS, macOS, and Android (TalkWaveformView + exact Compose port), replacing per-platform fakes: synthetic speaking pulses, constant speech-detected power, static Android bar rows, flat realtime listening. Listening/recording follow live mic levels on every route; agent speech follows the real playback envelope (AVAudioPlayer metering + playback-aligned PCM envelope + WebRTC stats); voice-note recording shows a live capture wave.
9327e49 to
b579a6a
Compare
|
Merged via squash.
|
…nclaw#102901) * feat(voice): universal talk waveform driven by real audio levels One shared Siri-style talk animation across iOS, watchOS, macOS, and Android (TalkWaveformView + exact Compose port), replacing per-platform fakes: synthetic speaking pulses, constant speech-detected power, static Android bar rows, flat realtime listening. Listening/recording follow live mic levels on every route; agent speech follows the real playback envelope (AVAudioPlayer metering + playback-aligned PCM envelope + WebRTC stats); voice-note recording shows a live capture wave. * fix(android): order waveform imports and sync native i18n inventory * chore(i18n): resync native inventory after rebase
…nclaw#102901) * feat(voice): universal talk waveform driven by real audio levels One shared Siri-style talk animation across iOS, watchOS, macOS, and Android (TalkWaveformView + exact Compose port), replacing per-platform fakes: synthetic speaking pulses, constant speech-detected power, static Android bar rows, flat realtime listening. Listening/recording follow live mic levels on every route; agent speech follows the real playback envelope (AVAudioPlayer metering + playback-aligned PCM envelope + WebRTC stats); voice-note recording shows a live capture wave. * fix(android): order waveform imports and sync native i18n inventory * chore(i18n): resync native inventory after rebase
1 participant
What Problem This Solves
Closes #102895.
Every platform had its own talk animation, and most of it was faked: iOS realtime routes rendered a flat wave (mic level never wired), agent speech pulsed on synthetic sine waves on iOS/macOS, Android's four voice visualizers were fully static hardcoded bar rows (while a real mic level already existed unused), watchOS had a static glyph, and voice-note recording showed no level feedback at all.
Why This Change Was Made
One canonical animation, real levels everywhere the platform allows:
TalkWaveformView(Siri-style wave) moves toOpenClawChatUIwith a phase contract —idle/thinking/listening(level:speechActive:)/speaking(level:).TalkWaveformMathis public and unit-tested;TalkWaveform.ktis an exact Compose port (same constants). The watch target compiles the same Swift file directly (it links no packages).PCMPlaybackEnvelope(OpenClawKit): playback-time-aligned envelope from PCM chunks streaming through the app (gateway TTS PCM, ElevenLabs PCM, realtime relay output), metered in 50 ms windows so network bursts don't distort the wave.AudioPlayerLevelMeter(OpenClawKit): AVAudioPlayer metering for buffered TTS clips (iOSTalkBufferedAudioPlayer, macOSTalkAudioPlayer, incl. MLX voice output).media-source/inbound-rtpaudioLevel).AVAudioRecordermetering on iOS/macOS;MediaRecorder.maxAmplitudepolling on Android; the shared recording row shows the live wave.AudioTrack(playback-paced), plusspeechActivefrom streaming user transcripts.TalkWaveRings/TalkOrbitArcs, all Android static bar rows.speaking(level: nil)means "no envelope available on this path" and renders the synthetic pulse; comments name the platform caps.User Impact
The talk screen (iOS), talk orb (macOS), voice/dictation screens (Android), watch voice turns, and the voice-note recording row now show one consistent waveform that actually follows your voice while recording/listening and the agent's audio while speaking, on all realtime routes.
Evidence
apps/shared/OpenClawKit:swift build+swift test— TalkWaveformMathTests, TalkAudioLevelTests, VoiceNoteRecorderTests (incl. new live-level test) all pass.apps/macos:swift buildclean; focused talk tests (TalkAudioPlayer|TalkModeRuntimeSpeech|TalkModeConfigParsing|LowCoverageViewSmoke|TalkModeGatewayConfig) pass — 22 tests, 5 suites.xcodebuild(OpenClaw scheme incl. the watch target) — BUILD SUCCEEDED. Note: the repo-wide SwiftFormat lint phase fails on current brew swiftformat 0.62.1 due to pre-existing drift in untouched files (LiveActivity, ShareExtension, chat transport, ...); every file in this diff lints clean under 0.62.1.:app:testPlayDebugUnitTestfor TalkWaveformMathTest, AudioLevelsTest, VoiceNoteRecorderControllerTest, VoiceScreenLogicTest, TalkModeManagerTest — BUILD SUCCESSFUL.docs/nodes/talk.mdmacOS overlay description.