Skip to content

Commit 5e0504a

Browse files
steipetesteipete-oai
steipete
and
steipete-oai
authored
feat(ui): simplify Talk settings (#100453)
Co-authored-by: Peter Steinberger <[email protected]>
1 parent 13e5902 commit 5e0504a

33 files changed

Lines changed: 687 additions & 479 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ Docs: https://docs.openclaw.ai
66

77
### Changes
88

9+
- **Control UI Talk controls:** keep voice, model, and sensitivity in the composer while moving provider, transport, VAD timing, and reasoning defaults to Settings → Communications → Talk.
910
- **Logbook work journal:** add a disabled-by-default bundled plugin that turns paired-node screen snapshots into a private timeline, daily standup, and timeline-grounded Q&A in a plugin-contributed Control UI tab. (#99930)
1011
- **Control UI message context:** reveal per-message token, context, and model details from the timestamp on hover or activation instead of showing a separate Context button.
1112
- **Control UI session titles:** reveal truncated recent-session names with a reduced-motion-safe hover animation.

‎docs/.generated/config-baseline.sha256‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
acdf418738f79d29f58cb6cfe5b7ab2d353cbcce2252861a4f527f85ea0c54af config-baseline.json
2-
4c98c716bc78e65c274ec374757357c1dcc9b5ec75c9e00ea4c20851531b7d1a config-baseline.core.json
3-
c68853362689981ac1cc1e55b9061286c2002104ff1c10bc44ee99a6080e169e config-baseline.channel.json
4-
859aa272b0dad53b7080c6fefcf775347ae79a1998ec39dd18b732c90d9df90c config-baseline.plugin.json
1+
ef4b1a8dd31f83262ac870f5e93f640f52be2813773a1e7dd7328366fae28cd0 config-baseline.json
2+
3063b8f84bc7c747fb4d2fc7976c8b7c1743c9de18c0c9a0e17f398ddd5951f6 config-baseline.core.json
3+
cb7503a2c1e489fa4d154dfa4bf4a9e2f1277233ab65fac71c1ba8dcb371a032 config-baseline.channel.json
4+
923843ceade95e79f0867f1c3a84732950eb95aa3323cd749dfbd1603325804e config-baseline.plugin.json

‎docs/gateway/config-agents.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1493,7 +1493,7 @@ Batches rapid text-only messages from the same sender into a single agent turn.
14931493

14941494
## Talk
14951495

1496-
Defaults for Talk mode (macOS/iOS/Android).
1496+
Defaults for Talk mode (macOS/iOS/Android and the browser Control UI).
14971497

14981498
```json5
14991499
{
@@ -1531,6 +1531,10 @@ Defaults for Talk mode (macOS/iOS/Android).
15311531
instructions: "Speak warmly and keep answers brief.",
15321532
mode: "realtime", // realtime | stt-tts | transcription
15331533
transport: "webrtc", // webrtc | provider-websocket | gateway-relay | managed-room
1534+
vadThreshold: 0.5,
1535+
silenceDurationMs: 500,
1536+
prefixPaddingMs: 300,
1537+
reasoningEffort: "medium",
15341538
brain: "agent-consult", // agent-consult | direct-tools | none
15351539
},
15361540
},
@@ -1550,6 +1554,10 @@ Defaults for Talk mode (macOS/iOS/Android).
15501554
- `speechLocale` sets the BCP 47 locale id used by iOS/macOS Talk speech recognition. Leave unset to use the device default.
15511555
- `silenceTimeoutMs` controls how long Talk mode waits after user silence before it sends the transcript. Unset keeps the platform default pause window (`700 ms on macOS and Android, 900 ms on iOS`).
15521556
- `realtime.instructions` appends provider-facing system instructions to OpenClaw's built-in realtime prompt, so voice style can be configured without losing default `openclaw_agent_consult` guidance.
1557+
- `realtime.vadThreshold` sets the provider voice-activity threshold from `0` (most sensitive) to `1` (least sensitive). Unset keeps the provider default.
1558+
- `realtime.silenceDurationMs` sets the positive whole-number silence window before the provider commits a realtime user turn. Unset keeps the provider default.
1559+
- `realtime.prefixPaddingMs` sets the non-negative whole-number amount of audio retained before detected speech begins. Unset keeps the provider default.
1560+
- `realtime.reasoningEffort` sets the provider-specific reasoning level for realtime sessions. Unset keeps the provider default.
15531561
- `realtime.consultRouting`: `"provider-direct"` (default) preserves direct provider replies when the realtime provider produces a final user transcript without `openclaw_agent_consult`. `"force-agent-consult"` routes the finalized request through OpenClaw instead.
15541562

15551563
---

‎docs/web/control-ui.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ The terminal is also available as a full-screen, terminal-only document at `/?vi
234234
<Accordion title="Talk mode (browser realtime)">
235235
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key auth profile, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`; OpenAI OAuth profiles do not configure Realtime voice. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. The browser never receives a standard provider API key: OpenAI receives an ephemeral Realtime client secret for WebRTC, and Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
236236

237-
The Chat composer includes a Talk options button next to the Talk start/stop button. Options apply to the next Talk session and can override provider, transport, model, voice, reasoning effort, VAD threshold, silence duration, and prefix padding. A blank option falls back to configured defaults or the provider default. Selecting Gateway relay forces the backend relay path; selecting WebRTC keeps the session client-owned and fails instead of silently falling back to relay if the provider cannot create a browser session.
237+
The Chat composer includes a Talk options button next to the Talk start/stop button. Its compact panel keeps only Voice, Model, and Sensitivity for the next Talk session. **More in Settings** opens **Settings → Communications → Talk**, where persistent provider, transport, reasoning effort, exact VAD threshold, silence duration, and prefix padding defaults live; changing those defaults requires `operator.admin` access. Blank composer values fall back to those configured defaults or the provider default. Configuring Gateway relay forces the backend relay path; configuring WebRTC keeps the session client-owned and fails instead of silently falling back to relay if the provider cannot create a browser session.
238238

239239
The Talk control itself is the waves button next to the microphone dictation button. When Talk starts, the composer status row shows `Connecting Talk...`, then `Talk live` while audio is connected, or `Asking OpenClaw...` while a realtime tool call is consulting the configured larger model through `talk.client.toolCall`.
240240

‎extensions/google/realtime-voice-provider.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -374,9 +374,13 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
374374
providerConfig: {
375375
apiKey: "gemini-key",
376376
model: "gemini-live-2.5-flash-preview",
377+
prefixPaddingMs: 100,
378+
silenceDurationMs: 300,
377379
voice: "Puck",
378380
temperature: 0.4,
379381
},
382+
prefixPaddingMs: 250,
383+
silenceDurationMs: 650,
380384
instructions: "Speak briefly.",
381385
tools: [
382386
{
@@ -399,6 +403,12 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
399403
config?: {
400404
liveConnectConstraints?: {
401405
config?: {
406+
realtimeInputConfig?: {
407+
automaticActivityDetection?: {
408+
prefixPaddingMs?: number;
409+
silenceDurationMs?: number;
410+
};
411+
};
402412
responseModalities?: string[];
403413
speechConfig?: { voiceConfig?: { prebuiltVoiceConfig?: { voiceName?: string } } };
404414
systemInstruction?: string;
@@ -423,6 +433,12 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
423433
expect(liveConstraints?.config?.responseModalities).toEqual(["AUDIO"]);
424434
expect(liveConstraints?.config?.temperature).toBe(0.4);
425435
expect(liveConstraints?.config?.systemInstruction).toBe("Speak briefly.");
436+
expect(
437+
liveConstraints?.config?.realtimeInputConfig?.automaticActivityDetection?.prefixPaddingMs,
438+
).toBe(250);
439+
expect(
440+
liveConstraints?.config?.realtimeInputConfig?.automaticActivityDetection?.silenceDurationMs,
441+
).toBe(650);
426442
expect(liveConstraints?.config?.speechConfig?.voiceConfig?.prebuiltVoiceConfig?.voiceName).toBe(
427443
"Puck",
428444
);

‎extensions/google/realtime-voice-provider.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -848,7 +848,14 @@ function decodeMulawSample(value: number): number {
848848
async function createGoogleRealtimeBrowserSession(
849849
req: RealtimeVoiceBrowserSessionCreateRequest,
850850
): Promise<RealtimeVoiceBrowserSession> {
851-
const config = normalizeProviderConfig(req.providerConfig);
851+
const providerConfig = normalizeProviderConfig(req.providerConfig);
852+
const prefixPaddingMs = asNonNegativeInteger(req.prefixPaddingMs);
853+
const silenceDurationMs = asNonNegativeInteger(req.silenceDurationMs);
854+
const config = {
855+
...providerConfig,
856+
...(prefixPaddingMs !== undefined ? { prefixPaddingMs } : {}),
857+
...(silenceDurationMs !== undefined ? { silenceDurationMs } : {}),
858+
};
852859
const apiKey = config.apiKey || resolveEnvApiKey();
853860
if (!apiKey) {
854861
throw new Error("Google Gemini API key missing");

‎packages/gateway-protocol/src/index.test.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -322,7 +322,12 @@ describe("validateTalkConfigResult", () => {
322322
instructions: "Speak with crisp diction.",
323323
mode: "realtime",
324324
transport: "gateway-relay",
325+
vadThreshold: 0.45,
326+
silenceDurationMs: 650,
327+
prefixPaddingMs: 250,
328+
reasoningEffort: "low",
325329
brain: "agent-consult",
330+
consultRouting: "force-agent-consult",
326331
},
327332
},
328333
},

‎packages/gateway-protocol/src/schema/channels.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -620,7 +620,14 @@ const TalkRealtimeConfigSchema = Type.Object(
620620
instructions: Type.Optional(Type.String()),
621621
mode: Type.Optional(TalkModeSchema),
622622
transport: Type.Optional(TalkTransportSchema),
623+
vadThreshold: Type.Optional(Type.Number({ minimum: 0, maximum: 1 })),
624+
silenceDurationMs: Type.Optional(Type.Integer({ minimum: 1 })),
625+
prefixPaddingMs: Type.Optional(Type.Integer({ minimum: 0 })),
626+
reasoningEffort: Type.Optional(Type.String({ minLength: 1 })),
623627
brain: Type.Optional(TalkBrainSchema),
628+
consultRouting: Type.Optional(
629+
Type.Union([Type.Literal("provider-direct"), Type.Literal("force-agent-consult")]),
630+
),
624631
},
625632
{ additionalProperties: false },
626633
);

‎src/config/schema.help.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,14 @@ export const FIELD_HELP: Record<string, string> = {
201201
"talk.realtime.mode": "Talk execution mode: realtime, stt-tts, or transcription.",
202202
"talk.realtime.transport":
203203
"Talk byte/session transport: webrtc, provider-websocket, gateway-relay, or managed-room.",
204+
"talk.realtime.vadThreshold":
205+
"Realtime voice activity detection threshold from 0 (most sensitive) to 1 (least sensitive).",
206+
"talk.realtime.silenceDurationMs":
207+
"Milliseconds of silence before a realtime Talk user turn is committed.",
208+
"talk.realtime.prefixPaddingMs":
209+
"Milliseconds of audio retained before realtime voice activity is detected.",
210+
"talk.realtime.reasoningEffort":
211+
"Provider-specific reasoning effort for realtime Talk sessions, such as minimal, low, medium, or high.",
204212
"talk.realtime.brain":
205213
"Talk reasoning strategy: agent-consult for Gateway-mediated agent help, direct-tools for local tool calls, or none.",
206214
"talk.realtime.consultRouting":

‎src/config/schema.labels.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1031,6 +1031,10 @@ export const FIELD_LABELS: Record<string, string> = {
10311031
"talk.realtime.instructions": "Talk Realtime Instructions",
10321032
"talk.realtime.mode": "Talk Realtime Mode",
10331033
"talk.realtime.transport": "Talk Realtime Transport",
1034+
"talk.realtime.vadThreshold": "Talk Realtime VAD Threshold",
1035+
"talk.realtime.silenceDurationMs": "Talk Realtime Silence Duration (ms)",
1036+
"talk.realtime.prefixPaddingMs": "Talk Realtime Prefix Padding (ms)",
1037+
"talk.realtime.reasoningEffort": "Talk Realtime Reasoning Effort",
10341038
"talk.realtime.brain": "Talk Realtime Brain",
10351039
"talk.realtime.consultRouting": "Talk Realtime Consult Routing",
10361040
channels: "Channels",

0 commit comments

Comments
 (0)