Skip to content

Commit 79dd65e

Browse files
scoootscooobscoootscooob
authored
feat(voice-call): improve realtime Meet voice agent
* feat(voice-call): inject agent context into realtime voice * fix(voice-call): stabilize realtime meet audio * fix(voice-call): delegate realtime consults to agent * Improve realtime Meet voice consult routing * Pin voice consult delivery to call session * Move voice changelog entries to changes * fix(voice-call): isolate final realtime transcripts * test(voice-call): trim redundant realtime coverage
1 parent 782963a commit 79dd65e

35 files changed

Lines changed: 2085 additions & 134 deletions

‎CHANGELOG.md‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,10 @@ Docs: https://docs.openclaw.ai
44

55
## Unreleased
66

7-
### Highlights
8-
9-
- Google Meet/Voice Call: make Twilio dial-in joins speak through the realtime Gemini voice bridge with paced audio streaming, backpressure-aware buffering, barge-in queue clearing, and no TwiML fallback during realtime speech, giving Meet participants a much snappier OpenClaw voice agent. (#77064) Thanks @scoootscooob.
10-
117
### Changes
128

9+
- Google Meet/Voice Call: make Twilio dial-in joins speak through the realtime Gemini voice bridge with paced audio streaming, backpressure-aware buffering, barge-in queue clearing, same-session agent consult routing, duplicate-consult coalescing, and no TwiML fallback during realtime speech, giving Meet participants a much snappier OpenClaw voice agent. (#77064) Thanks @scoootscooob.
10+
- Voice Call/realtime: add opt-in OpenClaw agent voice context capsules and consult-cadence guidance so Gemini/OpenAI realtime calls can sound like the configured agent without consulting the full agent on every ordinary turn. Thanks @scoootscooob.
1311
- Docker/Gateway: harden the gateway container by dropping `NET_RAW` and `NET_ADMIN` capabilities and enabling `no-new-privileges` in the bundled `docker-compose.yml`. Thanks @VintageAyu.
1412
- Telegram: accept plugin-owned numeric forum-topic targets in the agent message tool and keep reply-dispatch provider chunks behind a real stable runtime alias during in-place package updates. Fixes #77137. Thanks @richardmqq.
1513
- Channels/WhatsApp: support explicit WhatsApp Channel/Newsletter `@newsletter` outbound message targets with channel session metadata instead of DM routing. Fixes #13417; carries forward the narrow outbound target idea from #13424. Thanks @vincentkoc and @agentz-manfred.

‎docs/.generated/config-baseline.sha256‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
657060e80f3dc4b7d992e8625d2a8b0ff9b1b408960148d3f5f6a381d602359a config-baseline.json
1+
c93176f87a1e4576f5951b82037394c4bc9628bb6e056b6b24f96e662d6d636c config-baseline.json
22
92cbb12ca382f7424e7bd52df21798b10a57621f5c266909fa74e23f6cb973d7 config-baseline.core.json
33
cd7c0c7fb1435bc7e59099e9ac334462d5ad444016e9ab4512aae63a238f78dc config-baseline.channel.json
4-
9832b30a696930a3da7efccf38073137571e1b66cae84e54d747b733fdafcc54 config-baseline.plugin.json
4+
6871e789b74722e4ff2c877940dac256c232433ae26b305fc6ca782b90662097 config-baseline.plugin.json

‎docs/plugins/voice-call.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,8 @@ Current runtime behaviour:
229229
- Bundled realtime voice providers: Google Gemini Live (`google`) and OpenAI (`openai`), registered by their provider plugins.
230230
- Provider-owned raw config lives under `realtime.providers.<providerId>`.
231231
- Voice Call exposes the shared `openclaw_agent_consult` realtime tool by default. The realtime model can call it when the caller asks for deeper reasoning, current information, or normal OpenClaw tools.
232+
- `realtime.consultPolicy` optionally adds guidance for when the realtime model should call `openclaw_agent_consult`.
233+
- `realtime.agentContext.enabled` is default-off. When enabled, Voice Call injects a bounded agent identity, system prompt override, and selected workspace-file capsule into the realtime provider instructions at session setup.
232234
- `realtime.fastContext.enabled` is default-off. When enabled, Voice Call first searches indexed memory/session context for the consult question and returns those snippets to the realtime model within `realtime.fastContext.timeoutMs` before falling back to the full consult agent only if `realtime.fastContext.fallbackToConsult` is true.
233235
- If `realtime.provider` points at an unregistered provider, or no realtime voice provider is registered at all, Voice Call logs a warning and skips realtime media instead of failing the whole plugin.
234236
- Consult session keys reuse the stored call session when available, then fall back to the configured `sessionScope` (`per-phone` by default, or `per-call` for isolated calls).
@@ -243,6 +245,51 @@ Current runtime behaviour:
243245
| `owner` | Expose the consult tool and let the regular agent use the normal agent tool policy. |
244246
| `none` | Do not expose the consult tool. Custom `realtime.tools` are still passed through to the realtime provider. |
245247

248+
`realtime.consultPolicy` controls only the realtime model instructions:
249+
250+
| Policy | Guidance |
251+
| ------------- | ----------------------------------------------------------------------------------------------- |
252+
| `auto` | Keep the default prompt and let the provider decide when to call the consult tool. |
253+
| `substantive` | Answer simple conversational glue directly and consult before facts, memory, tools, or context. |
254+
| `always` | Consult before every substantive answer. |
255+
256+
### Agent voice context
257+
258+
Enable `realtime.agentContext` when the voice bridge should sound like the
259+
configured OpenClaw agent without paying a full agent-consult round trip on
260+
ordinary turns. The context capsule is added once when the realtime session is
261+
created, so it does not add per-turn latency. Calls to
262+
`openclaw_agent_consult` still run the full OpenClaw agent and should be used
263+
for tool work, current information, memory lookups, or workspace state.
264+
265+
```json5
266+
{
267+
plugins: {
268+
entries: {
269+
"voice-call": {
270+
config: {
271+
agentId: "main",
272+
realtime: {
273+
enabled: true,
274+
provider: "google",
275+
toolPolicy: "safe-read-only",
276+
consultPolicy: "substantive",
277+
agentContext: {
278+
enabled: true,
279+
maxChars: 6000,
280+
includeIdentity: true,
281+
includeSystemPrompt: true,
282+
includeWorkspaceFiles: true,
283+
files: ["SOUL.md", "IDENTITY.md", "USER.md"],
284+
},
285+
},
286+
},
287+
},
288+
},
289+
},
290+
}
291+
```
292+
246293
### Realtime provider examples
247294

248295
<Tabs>
@@ -268,6 +315,8 @@ Current runtime behaviour:
268315
provider: "google",
269316
instructions: "Speak briefly. Call openclaw_agent_consult before using deeper tools.",
270317
toolPolicy: "safe-read-only",
318+
consultPolicy: "substantive",
319+
agentContext: { enabled: true },
271320
providers: {
272321
google: {
273322
apiKey: "${GEMINI_API_KEY}",

‎extensions/google-meet/index.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1256,6 +1256,7 @@ describe("google-meet plugin", () => {
12561256
dtmfSequence: "123456#",
12571257
logger: expect.objectContaining({ info: expect.any(Function) }),
12581258
message: "Say exactly: I'm here and listening.",
1259+
sessionKey: expect.stringMatching(/^voice:google-meet:meet_/),
12591260
});
12601261
});
12611262

‎extensions/google-meet/src/runtime.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ function nowIso(): string {
4141
return new Date().toISOString();
4242
}
4343

44+
function buildTwilioVoiceCallSessionKey(meetingSessionId: string): string {
45+
return `voice:google-meet:${meetingSessionId}`;
46+
}
47+
4448
export function normalizeMeetUrl(input: unknown): string {
4549
const raw = normalizeOptionalString(input);
4650
if (!raw) {
@@ -478,6 +482,10 @@ export class GoogleMeetRuntime {
478482
dialInNumber,
479483
dtmfSequence,
480484
logger: this.params.logger,
485+
...(request.requesterSessionKey
486+
? { requesterSessionKey: request.requesterSessionKey }
487+
: {}),
488+
sessionKey: buildTwilioVoiceCallSessionKey(session.id),
481489
message: isGoogleMeetTalkBackMode(mode)
482490
? (request.message ??
483491
this.params.config.voiceCall.introMessage ??
@@ -505,7 +513,7 @@ export class GoogleMeetRuntime {
505513
session.notes.push(
506514
this.params.config.voiceCall.enabled
507515
? dtmfSequence
508-
? "Twilio transport delegated the phone leg to the voice-call plugin, then sent configured DTMF after connect before speaking."
516+
? "Twilio transport delegated the phone leg to the voice-call plugin, then queued configured DTMF before realtime connect."
509517
: "Twilio transport delegated the call to the voice-call plugin without configured DTMF."
510518
: "Twilio transport is an explicit dial plan; voice-call delegation is disabled.",
511519
);

‎extensions/google-meet/src/voice-call-gateway.test.ts‎

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ describe("Google Meet voice-call gateway", () => {
2828
gatewayMocks.startGatewayClientWhenEventLoopReady.mockClear();
2929
});
3030

31-
it("starts Twilio Meet calls, sends delayed DTMF, then speaks the intro without TwiML fallback", async () => {
31+
it("starts Twilio Meet calls with pre-connect DTMF, then speaks the intro without TwiML fallback", async () => {
3232
const config = resolveGoogleMeetConfig({
3333
voiceCall: {
3434
gatewayUrl: "ws://127.0.0.1:18789",
@@ -43,6 +43,8 @@ describe("Google Meet voice-call gateway", () => {
4343
dialInNumber: "+15551234567",
4444
dtmfSequence: "123456#",
4545
message: "Say exactly: I'm here and listening.",
46+
requesterSessionKey: "agent:main:discord:channel:general",
47+
sessionKey: "voice:google-meet:meet-1",
4648
});
4749

4850
await join;
@@ -53,20 +55,14 @@ describe("Google Meet voice-call gateway", () => {
5355
{
5456
to: "+15551234567",
5557
mode: "conversation",
58+
dtmfSequence: "123456#",
59+
requesterSessionKey: "agent:main:discord:channel:general",
60+
sessionKey: "voice:google-meet:meet-1",
5661
},
5762
{ timeoutMs: 30_000 },
5863
);
5964
expect(gatewayMocks.request).toHaveBeenNthCalledWith(
6065
2,
61-
"voicecall.dtmf",
62-
{
63-
callId: "call-1",
64-
digits: "123456#",
65-
},
66-
{ timeoutMs: 30_000 },
67-
);
68-
expect(gatewayMocks.request).toHaveBeenNthCalledWith(
69-
3,
7066
"voicecall.speak",
7167
{
7268
callId: "call-1",
@@ -75,13 +71,12 @@ describe("Google Meet voice-call gateway", () => {
7571
},
7672
{ timeoutMs: 30_000 },
7773
);
78-
expect(gatewayMocks.request).toHaveBeenCalledTimes(3);
74+
expect(gatewayMocks.request).toHaveBeenCalledTimes(2);
7975
});
8076

8177
it("skips the intro without failing when the realtime bridge is not ready", async () => {
8278
gatewayMocks.request
8379
.mockResolvedValueOnce({ callId: "call-1" })
84-
.mockResolvedValueOnce({ success: true })
8580
.mockResolvedValueOnce({ success: false, error: "No active realtime bridge for call" });
8681
const config = resolveGoogleMeetConfig({
8782
voiceCall: {

‎extensions/google-meet/src/voice-call-gateway.ts‎

Lines changed: 27 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
12
import {
23
GatewayClient,
34
startGatewayClientWhenEventLoopReady,
@@ -18,11 +19,6 @@ type VoiceCallSpeakResult = {
1819
error?: string;
1920
};
2021

21-
type VoiceCallDtmfResult = {
22-
success?: boolean;
23-
error?: string;
24-
};
25-
2622
type VoiceCallMeetJoinResult = {
2723
callId: string;
2824
dtmfSent: boolean;
@@ -87,19 +83,24 @@ export async function joinMeetViaVoiceCallGateway(params: {
8783
dtmfSequence?: string;
8884
logger?: RuntimeLogger;
8985
message?: string;
86+
requesterSessionKey?: string;
87+
sessionKey?: string;
9088
}): Promise<VoiceCallMeetJoinResult> {
9189
let client: VoiceCallGatewayClient | undefined;
9290

9391
try {
9492
client = await createConnectedGatewayClient(params.config);
9593
params.logger?.info(
96-
`[google-meet] Delegating Twilio join to Voice Call (dtmf=${params.dtmfSequence ? "post-connect" : "none"}, intro=${params.message ? "delayed" : "none"})`,
94+
`[google-meet] Delegating Twilio join to Voice Call (dtmf=${params.dtmfSequence ? "pre-connect" : "none"}, intro=${params.message ? "delayed" : "none"})`,
9795
);
9896
const start = (await client.request(
9997
"voicecall.start",
10098
{
10199
to: params.dialInNumber,
102100
mode: "conversation",
101+
...(params.dtmfSequence ? { dtmfSequence: params.dtmfSequence } : {}),
102+
...(params.requesterSessionKey ? { requesterSessionKey: params.requesterSessionKey } : {}),
103+
...(params.sessionKey ? { sessionKey: params.sessionKey } : {}),
103104
},
104105
{ timeoutMs: params.config.voiceCall.requestTimeoutMs },
105106
)) as VoiceCallStartResult;
@@ -109,27 +110,10 @@ export async function joinMeetViaVoiceCallGateway(params: {
109110
params.logger?.info(
110111
`[google-meet] Voice Call Twilio phone leg started: callId=${start.callId}`,
111112
);
112-
let dtmfSent = false;
113-
if (params.dtmfSequence) {
114-
const delayMs = params.config.voiceCall.dtmfDelayMs;
115-
params.logger?.info(
116-
`[google-meet] Waiting ${delayMs}ms before sending Meet DTMF for callId=${start.callId}`,
117-
);
118-
await sleep(delayMs);
119-
const dtmf = (await client.request(
120-
"voicecall.dtmf",
121-
{
122-
callId: start.callId,
123-
digits: params.dtmfSequence,
124-
},
125-
{ timeoutMs: params.config.voiceCall.requestTimeoutMs },
126-
)) as VoiceCallDtmfResult;
127-
if (dtmf.success === false) {
128-
throw new Error(dtmf.error || "voicecall.dtmf failed");
129-
}
130-
dtmfSent = true;
113+
const dtmfSent = Boolean(params.dtmfSequence);
114+
if (dtmfSent) {
131115
params.logger?.info(
132-
`[google-meet] Meet DTMF sent after phone leg connected: callId=${start.callId} digits=${params.dtmfSequence.length}`,
116+
`[google-meet] Meet DTMF queued before realtime connect: callId=${start.callId} digits=${params.dtmfSequence?.length ?? 0}`,
133117
);
134118
}
135119
let introSent = false;
@@ -141,15 +125,23 @@ export async function joinMeetViaVoiceCallGateway(params: {
141125
);
142126
await sleep(delayMs);
143127
}
144-
const spoken = (await client.request(
145-
"voicecall.speak",
146-
{
147-
callId: start.callId,
148-
allowTwimlFallback: false,
149-
message: params.message,
150-
},
151-
{ timeoutMs: params.config.voiceCall.requestTimeoutMs },
152-
)) as VoiceCallSpeakResult;
128+
let spoken: VoiceCallSpeakResult;
129+
try {
130+
spoken = (await client.request(
131+
"voicecall.speak",
132+
{
133+
callId: start.callId,
134+
allowTwimlFallback: false,
135+
message: params.message,
136+
},
137+
{ timeoutMs: params.config.voiceCall.requestTimeoutMs },
138+
)) as VoiceCallSpeakResult;
139+
} catch (err) {
140+
params.logger?.warn?.(
141+
`[google-meet] Skipped intro speech because realtime bridge was not ready: ${formatErrorMessage(err)}`,
142+
);
143+
spoken = { success: false };
144+
}
153145
if (spoken.success === false) {
154146
params.logger?.warn?.(
155147
`[google-meet] Skipped intro speech because realtime bridge was not ready: ${

‎extensions/google/index.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,16 @@ import type {
33
ProviderReplaySessionEntry,
44
ProviderSanitizeReplayHistoryContext,
55
} from "openclaw/plugin-sdk/plugin-entry";
6+
import { createTestPluginApi } from "openclaw/plugin-sdk/plugin-test-api";
67
import {
78
registerProviderPlugin,
89
requireRegisteredProvider,
910
} from "openclaw/plugin-sdk/plugin-test-runtime";
1011
import { createCapturedThinkingConfigStream } from "openclaw/plugin-sdk/provider-test-contracts";
12+
import type { RealtimeVoiceProviderPlugin } from "openclaw/plugin-sdk/realtime-voice";
1113
import { describe, expect, it } from "vitest";
1214
import { registerGoogleGeminiCliProvider } from "./gemini-cli-provider.js";
15+
import googlePlugin from "./index.js";
1316
import { registerGoogleProvider } from "./provider-registration.js";
1417

1518
const googleProviderPlugin = {
@@ -226,4 +229,26 @@ describe("google provider plugin hooks", () => {
226229
expect(googleProvider.buildReplayPolicy).toBe(cliProvider.buildReplayPolicy);
227230
expect(googleProvider.wrapStreamFn).toBe(cliProvider.wrapStreamFn);
228231
});
232+
233+
it("buffers early realtime audio while the lazy Google bridge loads", () => {
234+
let realtimeProvider: RealtimeVoiceProviderPlugin | undefined;
235+
googlePlugin.register(
236+
createTestPluginApi({
237+
registerRealtimeVoiceProvider(provider) {
238+
realtimeProvider = provider;
239+
},
240+
}),
241+
);
242+
243+
const bridge = realtimeProvider?.createBridge({
244+
providerConfig: { apiKey: "gemini-key" },
245+
onAudio() {},
246+
onClearAudio() {},
247+
});
248+
249+
expect(bridge).toBeDefined();
250+
expect(() => bridge?.sendAudio(Buffer.alloc(160))).not.toThrow();
251+
expect(() => bridge?.setMediaTimestamp(20)).not.toThrow();
252+
expect(() => bridge?.sendUserMessage?.("hello")).not.toThrow();
253+
});
229254
});

0 commit comments

Comments
 (0)