Skip to content

Commit 07a7d59

Browse files
steipetesteipete
authored
feat(providers): add Qwen 3.6 Flash and Gemini 3.1 Live (#102791)
* feat(providers): add current Qwen and Gemini Live models * fix(google): align live thinking enum type * fix(google): align live thinking enum type
1 parent c8c1f37 commit 07a7d59

17 files changed

Lines changed: 387 additions & 78 deletions

‎docs/plugins/google-meet.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -736,7 +736,7 @@ An `openclaw doctor --fix` migration exists for the legacy `realtime.provider: "
736736
provider: "openai",
737737
transcriptionProvider: "openai",
738738
voiceProvider: "google",
739-
model: "gemini-2.5-flash-native-audio-preview-12-2025",
739+
model: "gemini-3.1-flash-live-preview",
740740
agentId: "jay",
741741
toolPolicy: "owner",
742742
introMessage: "Say exactly: I'm here.",

‎docs/plugins/voice-call.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -328,7 +328,7 @@ for tool work, current information, memory lookups, or workspace state.
328328
<Tabs>
329329
<Tab title="Google Gemini Live">
330330
Defaults: API key from `realtime.providers.google.apiKey`, `GEMINI_API_KEY`,
331-
or `GOOGLE_API_KEY`; model `gemini-2.5-flash-native-audio-preview-12-2025`;
331+
or `GOOGLE_API_KEY`; model `gemini-3.1-flash-live-preview`;
332332
voice `Kore`. `sessionResumption` and `contextWindowCompression` default on
333333
for longer, reconnectable calls. Use `silenceDurationMs`,
334334
`startSensitivity`, and `endSensitivity` to tune faster turn-taking on
@@ -355,7 +355,7 @@ for tool work, current information, memory lookups, or workspace state.
355355
providers: {
356356
google: {
357357
apiKey: "${GEMINI_API_KEY}",
358-
model: "gemini-2.5-flash-native-audio-preview-12-2025",
358+
model: "gemini-3.1-flash-live-preview",
359359
speakerVoice: "Kore",
360360
silenceDurationMs: 500,
361361
startSensitivity: "high",

‎docs/providers/google.md‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -342,14 +342,14 @@ Gemini Live API for backend audio bridges such as Voice Call and Google Meet.
342342

343343
| Setting | Config path | Default |
344344
| --------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
345-
| Model | `plugins.entries.voice-call.config.realtime.providers.google.model` | `gemini-2.5-flash-native-audio-preview-12-2025` |
345+
| Model | `plugins.entries.voice-call.config.realtime.providers.google.model` | `gemini-3.1-flash-live-preview` |
346346
| Voice | `...google.voice` | `Kore` |
347347
| Temperature | `...google.temperature` | (unset) |
348348
| VAD start sensitivity | `...google.startSensitivity` | (unset) |
349349
| VAD end sensitivity | `...google.endSensitivity` | (unset) |
350350
| Silence duration | `...google.silenceDurationMs` | (unset) |
351351
| Activity handling | `...google.activityHandling` | Google default, `start-of-activity-interrupts` |
352-
| Turn coverage | `...google.turnCoverage` | Google default, `only-activity` |
352+
| Turn coverage | `...google.turnCoverage` | Google default, `audio-activity-and-all-video` |
353353
| Disable auto VAD | `...google.automaticActivityDetectionDisabled` | `false` |
354354
| Session resumption | `...google.sessionResumption` | `true` |
355355
| Context compression | `...google.contextWindowCompression` | `true` |
@@ -369,10 +369,10 @@ Example Voice Call realtime config:
369369
provider: "google",
370370
providers: {
371371
google: {
372-
model: "gemini-2.5-flash-native-audio-preview-12-2025",
372+
model: "gemini-3.1-flash-live-preview",
373373
speakerVoice: "Kore",
374374
activityHandling: "start-of-activity-interrupts",
375-
turnCoverage: "only-activity",
375+
turnCoverage: "audio-activity-and-all-video",
376376
},
377377
},
378378
},
@@ -393,6 +393,15 @@ Gemini API transcription is enabled without `languageCodes`; the current Google
393393
SDK rejects language-code hints on this API path.
394394
</Note>
395395

396+
<Note>
397+
Gemini 3.1 Live accepts conversational text through realtime input and uses
398+
sequential function calling. OpenClaw omits the older `NON_BLOCKING`, function
399+
response scheduling, and affective-dialog fields for this model. Prefer
400+
`thinkingLevel`; configured positive `thinkingBudget` values are mapped to the
401+
nearest supported level, while `-1` leaves Google's default in place. See the
402+
[Gemini Live capability comparison](https://ai.google.dev/gemini-api/docs/live-api/capabilities).
403+
</Note>
404+
396405
<Note>
397406
Control UI Talk supports Google Live browser sessions with constrained one-use
398407
tokens. Backend-only realtime voice providers can also run through the generic

‎docs/providers/qwen.md‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,8 @@ Qwen Cloud is an official external OpenClaw provider plugin with canonical id `q
1717
| API style | OpenAI-compatible |
1818

1919
<Tip>
20-
`qwen3.7-plus` and `qwen3.6-plus` work with Coding Plan and Standard endpoints. For `qwen3.7-max`, use a **Standard (pay-as-you-go)** endpoint.
20+
`qwen3.7-plus` and `qwen3.6-plus` work with Coding Plan and Standard endpoints.
21+
For `qwen3.7-max` or `qwen3.6-flash`, use a **Standard (pay-as-you-go)** endpoint.
2122
</Tip>
2223

2324
## Install plugin
@@ -84,7 +85,7 @@ Choose your plan type and follow the setup steps.
8485
</Tab>
8586

8687
<Tab title="Standard (pay-as-you-go)">
87-
**Best for:** pay-as-you-go access through the Standard Model Studio endpoint, including `qwen3.7-max`, which is not available on the Coding Plan.
88+
**Best for:** pay-as-you-go access through the Standard Model Studio endpoint, including `qwen3.7-max` and `qwen3.6-flash`, which are not available on the Coding Plan.
8889

8990
<Steps>
9091
<Step title="Get your API key">
@@ -198,6 +199,7 @@ Plan configs omit models that only work on the Standard endpoint.
198199
| Model ref | Input | Context | Notes |
199200
| --------------------------- | ----------- | --------- | ----------------------- |
200201
| `qwen/qwen3.5-plus` | text, image | 1,000,000 | Default model |
202+
| `qwen/qwen3.6-flash` | text, image | 1,000,000 | Standard endpoints only |
201203
| `qwen/qwen3.6-plus` | text, image | 1,000,000 | Coding Plan + Standard |
202204
| `qwen/qwen3.7-max` | text | 1,000,000 | Standard endpoints only |
203205
| `qwen/qwen3.7-plus` | text, image | 1,000,000 | Coding Plan + Standard |
@@ -217,7 +219,8 @@ present in the static catalog.
217219

218220
## Thinking controls
219221

220-
`qwen3.7-max`, `qwen3.7-plus`, `qwen3.6-plus`, and `qwen/MiniMax-M2.5` are
222+
`qwen3.7-max`, `qwen3.7-plus`, `qwen3.6-flash`, `qwen3.6-plus`, and
223+
`qwen/MiniMax-M2.5` are
221224
reasoning-enabled in the built-in catalog. For reasoning models on the `qwen`
222225
family, the provider maps OpenClaw thinking levels to DashScope's top-level
223226
`enable_thinking` request flag: disabled thinking sends `enable_thinking: false`,
@@ -263,15 +266,15 @@ See [Video generation](/tools/video-generation) for shared tool parameters, prov
263266
## Advanced configuration
264267

265268
<AccordionGroup>
266-
<Accordion title="Qwen 3.7 availability">
267-
`qwen3.7-plus` is available on Coding Plan and Standard endpoints. `qwen3.7-max` is Standard-only. The Standard (pay-as-you-go) endpoints are:
269+
<Accordion title="Qwen 3.6 and 3.7 availability">
270+
`qwen3.7-plus` and `qwen3.6-plus` are available on Coding Plan and Standard endpoints. `qwen3.7-max` and `qwen3.6-flash` are Standard-only. The Standard (pay-as-you-go) endpoints are:
268271

269272
- China: `dashscope.aliyuncs.com/compatible-mode/v1`
270273
- Global: `dashscope-intl.aliyuncs.com/compatible-mode/v1`
271274

272-
OpenClaw omits `qwen3.7-max` from Coding Plan catalogs. If a Coding Plan
273-
endpoint returns an "unsupported model" error for it, switch to the matching
274-
Standard endpoint and key.
275+
OpenClaw omits `qwen3.7-max` and `qwen3.6-flash` from Coding Plan catalogs.
276+
If a Coding Plan endpoint returns an "unsupported model" error for either,
277+
switch to the matching Standard endpoint and key.
275278

276279
</Accordion>
277280

‎extensions/google/index.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -404,6 +404,7 @@ describe("google provider plugin hooks", () => {
404404
if (!bridge) {
405405
throw new Error("expected Google realtime bridge");
406406
}
407+
expect(bridge.supportsToolResultContinuation).toBe(false);
407408
expect(bridge.sendAudio(Buffer.alloc(160))).toBeUndefined();
408409
expect(bridge.setMediaTimestamp(20)).toBeUndefined();
409410
expect(bridge.sendUserMessage?.("hello")).toBeUndefined();

‎extensions/google/index.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -245,7 +245,9 @@ function createLazyGoogleRealtimeVoiceBridge(
245245
}
246246
};
247247
return {
248-
supportsToolResultContinuation: true,
248+
get supportsToolResultContinuation() {
249+
return bridge?.supportsToolResultContinuation ?? false;
250+
},
249251
connect: async () => {
250252
const loadedBridge = await loadBridge();
251253
if (closed) {

‎extensions/google/realtime-voice-provider.test.ts‎

Lines changed: 153 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,7 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
154154
it("declares realtime Talk capabilities for catalog selection", () => {
155155
const provider = buildGoogleRealtimeVoiceProvider();
156156

157+
expect(provider.defaultModel).toBe("gemini-3.1-flash-live-preview");
157158
expect(provider.capabilities).toEqual({
158159
transports: ["provider-websocket", "gateway-relay"],
159160
inputAudioFormats: [
@@ -173,6 +174,35 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
173174
});
174175
});
175176

177+
it("uses Gemini 3.1 Live-compatible defaults", async () => {
178+
const provider = buildGoogleRealtimeVoiceProvider();
179+
const bridge = provider.createBridge({
180+
providerConfig: {
181+
apiKey: "gemini-key",
182+
enableAffectiveDialog: true,
183+
thinkingBudget: 8_193,
184+
},
185+
tools: [createRealtimeTool("openclaw_agent_consult")],
186+
onAudio: vi.fn(),
187+
onClearAudio: vi.fn(),
188+
});
189+
190+
expect(bridge.supportsToolResultContinuation).toBe(false);
191+
await bridge.connect();
192+
193+
const params = lastConnectParams();
194+
expect(params.model).toBe("gemini-3.1-flash-live-preview");
195+
expect(params.config.thinkingConfig).toEqual({ thinkingLevel: "HIGH" });
196+
expect(params.config).not.toHaveProperty("enableAffectiveDialog");
197+
const config = params.config as {
198+
tools?: Array<{ functionDeclarations?: Array<{ behavior?: string; name?: string }> }>;
199+
};
200+
expect(config.tools?.[0]?.functionDeclarations?.[0]).toMatchObject({
201+
name: "openclaw_agent_consult",
202+
});
203+
expect(config.tools?.[0]?.functionDeclarations?.[0]).not.toHaveProperty("behavior");
204+
});
205+
176206
it("normalizes provider config and cfg model-provider key fallback", () => {
177207
const provider = buildGoogleRealtimeVoiceProvider();
178208
const resolved = provider.resolveConfig?.({
@@ -416,6 +446,7 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
416446
const bridge = provider.createBridge({
417447
providerConfig: {
418448
apiKey: "gemini-key",
449+
model: "gemini-live-2.5-flash-preview",
419450
thinkingBudget: -1,
420451
},
421452
onAudio: vi.fn(),
@@ -427,6 +458,22 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
427458
expect(lastConnectParams().config.thinkingConfig).toEqual({ thinkingBudget: -1 });
428459
});
429460

461+
it("omits adaptive thinking budgets for Gemini 3.1 Live", async () => {
462+
const provider = buildGoogleRealtimeVoiceProvider();
463+
const bridge = provider.createBridge({
464+
providerConfig: {
465+
apiKey: "gemini-key",
466+
thinkingBudget: -1,
467+
},
468+
onAudio: vi.fn(),
469+
onClearAudio: vi.fn(),
470+
});
471+
472+
await bridge.connect();
473+
474+
expect(lastConnectParams().config).not.toHaveProperty("thinkingConfig");
475+
});
476+
430477
it("creates constrained browser sessions for Google Live Talk", async () => {
431478
const provider = buildGoogleRealtimeVoiceProvider();
432479

@@ -546,6 +593,46 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
546593
]);
547594
});
548595

596+
it("constrains default browser sessions to Gemini 3.1 capabilities", async () => {
597+
const provider = buildGoogleRealtimeVoiceProvider();
598+
599+
const sessionLocal = await provider.createBrowserSession?.({
600+
providerConfig: {
601+
apiKey: "gemini-key",
602+
enableAffectiveDialog: true,
603+
thinkingLevel: "low",
604+
thinkingBudget: 8_193,
605+
},
606+
tools: [createRealtimeTool("openclaw_agent_consult")],
607+
});
608+
609+
const tokenConfig = requireFirstMockArg(createTokenMock, "Google Live auth token config") as {
610+
config?: {
611+
liveConnectConstraints?: {
612+
config?: {
613+
enableAffectiveDialog?: boolean;
614+
thinkingConfig?: unknown;
615+
tools?: Array<{
616+
functionDeclarations?: Array<{ behavior?: string; name?: string }>;
617+
}>;
618+
};
619+
model?: string;
620+
};
621+
};
622+
};
623+
const constraints = tokenConfig.config?.liveConnectConstraints;
624+
expect(constraints?.model).toBe("gemini-3.1-flash-live-preview");
625+
expect(constraints?.config?.thinkingConfig).toEqual({ thinkingLevel: "LOW" });
626+
expect(constraints?.config).not.toHaveProperty("enableAffectiveDialog");
627+
expect(constraints?.config?.tools?.[0]?.functionDeclarations?.[0]).toMatchObject({
628+
name: "openclaw_agent_consult",
629+
});
630+
expect(constraints?.config?.tools?.[0]?.functionDeclarations?.[0]).not.toHaveProperty(
631+
"behavior",
632+
);
633+
expect(sessionLocal?.model).toBe("gemini-3.1-flash-live-preview");
634+
});
635+
549636
it("rejects browser session expiry outside Date range", async () => {
550637
vi.spyOn(Date, "now").mockReturnValue(8_640_000_000_000_001);
551638
const provider = buildGoogleRealtimeVoiceProvider();
@@ -907,7 +994,7 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
907994
expect(config.realtimeInputConfig?.automaticActivityDetection?.disabled).toBe(true);
908995
});
909996

910-
it("sends text prompts as ordered client turns", async () => {
997+
it("sends Gemini 3.1 text prompts as realtime input", async () => {
911998
const provider = buildGoogleRealtimeVoiceProvider();
912999
const bridge = provider.createBridge({
9131000
providerConfig: { apiKey: "gemini-key" },
@@ -921,6 +1008,27 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
9211008

9221009
bridge.sendUserMessage?.(" Say hello. ");
9231010

1011+
expect(session.sendRealtimeInput).toHaveBeenCalledWith({ text: "Say hello." });
1012+
expect(session.sendClientContent).not.toHaveBeenCalled();
1013+
});
1014+
1015+
it("keeps ordered client turns for explicit Gemini 2.5 sessions", async () => {
1016+
const provider = buildGoogleRealtimeVoiceProvider();
1017+
const bridge = provider.createBridge({
1018+
providerConfig: {
1019+
apiKey: "gemini-key",
1020+
model: "gemini-live-2.5-flash-preview",
1021+
},
1022+
onAudio: vi.fn(),
1023+
onClearAudio: vi.fn(),
1024+
});
1025+
1026+
await bridge.connect();
1027+
lastConnectParams().callbacks.onopen();
1028+
lastConnectParams().callbacks.onmessage({ setupComplete: { sessionId: "session-1" } });
1029+
1030+
bridge.sendUserMessage?.(" Say hello. ");
1031+
9241032
expect(session.sendClientContent).toHaveBeenCalledWith({
9251033
turns: [{ role: "user", parts: [{ text: "Say hello." }] }],
9261034
turnComplete: true,
@@ -1203,7 +1311,10 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
12031311
it("keeps Google Live consult calls open after continuing tool responses", async () => {
12041312
const provider = buildGoogleRealtimeVoiceProvider();
12051313
const bridge = provider.createBridge({
1206-
providerConfig: { apiKey: "gemini-key" },
1314+
providerConfig: {
1315+
apiKey: "gemini-key",
1316+
model: "gemini-live-2.5-flash-preview",
1317+
},
12071318
onAudio: vi.fn(),
12081319
onClearAudio: vi.fn(),
12091320
onToolCall: vi.fn(),
@@ -1249,6 +1360,46 @@ describe("buildGoogleRealtimeVoiceProvider", () => {
12491360
});
12501361
});
12511362

1363+
it("keeps Gemini 3.1 consult calls pending after rejecting continuation", async () => {
1364+
const provider = buildGoogleRealtimeVoiceProvider();
1365+
const onError = vi.fn();
1366+
const bridge = provider.createBridge({
1367+
providerConfig: { apiKey: "gemini-key" },
1368+
onAudio: vi.fn(),
1369+
onClearAudio: vi.fn(),
1370+
onToolCall: vi.fn(),
1371+
onError,
1372+
});
1373+
1374+
await bridge.connect();
1375+
lastConnectParams().callbacks.onmessage({
1376+
setupComplete: { sessionId: "session-1" },
1377+
toolCall: {
1378+
functionCalls: [
1379+
{ id: "consult-call", name: "openclaw_agent_consult", args: { prompt: "hi" } },
1380+
],
1381+
},
1382+
});
1383+
1384+
bridge.submitToolResult("consult-call", { status: "working" }, { willContinue: true });
1385+
expect(session.sendToolResponse).not.toHaveBeenCalled();
1386+
expect(requireFirstError(onError).message).toContain(
1387+
"does not support continuing tool responses",
1388+
);
1389+
1390+
bridge.submitToolResult("consult-call", { text: "The meeting starts at 3." });
1391+
1392+
expect(session.sendToolResponse).toHaveBeenCalledWith({
1393+
functionResponses: [
1394+
{
1395+
id: "consult-call",
1396+
name: "openclaw_agent_consult",
1397+
response: { text: "The meeting starts at 3." },
1398+
},
1399+
],
1400+
});
1401+
});
1402+
12521403
it("does not send malformed Live API tool responses without a matching call name", async () => {
12531404
const provider = buildGoogleRealtimeVoiceProvider();
12541405
const onError = vi.fn();

0 commit comments

Comments
 (0)