Skip to content

Commit d7d19a3

Browse files
steipetesteipete-oai
steipete
and
steipete-oai
authored
fix(openai): reuse Codex auth for realtime voice (#100671)
Co-authored-by: Peter Steinberger <[email protected]>
1 parent 92ceaa2 commit d7d19a3

7 files changed

Lines changed: 208 additions & 88 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Docs: https://docs.openclaw.ai
2222

2323
### Fixes
2424

25+
- **OpenAI Realtime Codex auth:** reuse external Codex OAuth profiles for Realtime voice sessions when no explicit OpenAI API key is configured.
2526
- **Android Canvas navigation:** block device-local loopback and unspecified main-frame web targets across direct, user, JavaScript, and redirect navigation while preserving remote, LAN, emulator-host, and bundled canvases. (#99874) Thanks @ly85206559.
2627
- **Android network recovery:** reconnect Gateway sessions immediately when Android regains a validated network instead of waiting for the current reconnect backoff. (#100347) Thanks @ly85206559.
2728
- **Android Voice layout:** keep Voice settings controls within their intended width so nested cards do not overflow or clip on constrained screens. (#100491) Thanks @IWhatsskill.

‎docs/providers/openai.md‎

Lines changed: 30 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -90,37 +90,37 @@ were not set by explicit runtime config.
9090

9191
## OpenClaw feature coverage
9292

93-
| OpenAI capability | OpenClaw surface | Status |
94-
| ------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
95-
| Chat / Responses | `openai/<model>` model provider | Yes |
96-
| Codex subscription models | `openai/<model>` with OpenAI OAuth | Yes |
97-
| Legacy Codex model refs | old Codex model refs, `codex-cli/<model>` | Repaired by doctor to `openai/<model>` |
98-
| Codex app-server harness | `openai/<model>` with runtime unset, or provider/model `agentRuntime.id: codex` | Yes |
99-
| Server-side web search | Native OpenAI Responses tool | Yes, when web search is enabled and no other provider is pinned |
100-
| Images | `image_generate` | Yes |
101-
| Videos | `video_generate` | Yes |
102-
| Text-to-speech | `messages.tts.provider: "openai"` / `tts` | Yes |
103-
| Batch speech-to-text | `tools.media.audio` / media understanding | Yes |
104-
| Streaming speech-to-text | Voice Call `streaming.provider: "openai"` | Yes |
105-
| Realtime voice | Voice Call `realtime.provider: "openai"` / Control UI Talk `talk.realtime.provider: "openai"` | Yes (needs OpenAI Platform credits, not Codex/ChatGPT subscription) |
106-
| Embeddings | memory embedding provider | Yes |
93+
| OpenAI capability | OpenClaw surface | Status |
94+
| ------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
95+
| Chat / Responses | `openai/<model>` model provider | Yes |
96+
| Codex subscription models | `openai/<model>` with OpenAI OAuth | Yes |
97+
| Legacy Codex model refs | old Codex model refs, `codex-cli/<model>` | Repaired by doctor to `openai/<model>` |
98+
| Codex app-server harness | `openai/<model>` with runtime unset, or provider/model `agentRuntime.id: codex` | Yes |
99+
| Server-side web search | Native OpenAI Responses tool | Yes, when web search is enabled and no other provider is pinned |
100+
| Images | `image_generate` | Yes |
101+
| Videos | `video_generate` | Yes |
102+
| Text-to-speech | `messages.tts.provider: "openai"` / `tts` | Yes |
103+
| Batch speech-to-text | `tools.media.audio` / media understanding | Yes |
104+
| Streaming speech-to-text | Voice Call `streaming.provider: "openai"` | Yes |
105+
| Realtime voice | Voice Call `realtime.provider: "openai"` / Control UI Talk `talk.realtime.provider: "openai"` | Yes (OpenAI API key or Codex OAuth) |
106+
| Embeddings | memory embedding provider | Yes |
107107

108108
<Note>
109109
OpenAI Realtime voice goes through the public **OpenAI Platform Realtime
110-
API**, billed against OpenAI Platform credits rather than Codex/ChatGPT
111-
subscription quota. An account whose Codex-backed chat models run fine on
112-
OAuth still needs a Platform API key with funded billing for Realtime voice.
110+
API**. It accepts either a Platform API key or an `openai` OAuth profile,
111+
including an automatically discovered external Codex login. API-key sessions
112+
use the key's Platform billing; OAuth availability and billing follow the
113+
authenticated account's Realtime entitlement.
113114

114-
Fix: top up Platform credits at
115+
If API-key auth reports missing billing, top up Platform credits at
115116
[platform.openai.com/account/billing](https://platform.openai.com/account/billing)
116-
for the organization backing your realtime credentials. Realtime voice accepts
117-
the `openai` API-key auth profile created by
118-
`openclaw onboard --auth-choice openai-api-key`, a Platform `OPENAI_API_KEY`
119-
set via `talk.realtime.providers.openai.apiKey` for Control UI Talk or
117+
for the organization backing your realtime credentials when using API-key
118+
auth. Realtime voice accepts the `openai` API-key auth profile created by
119+
`openclaw onboard --auth-choice openai-api-key`, an `openai` OAuth profile or
120+
external Codex login, a Platform `OPENAI_API_KEY` set via
121+
`talk.realtime.providers.openai.apiKey` for Control UI Talk, or
120122
`plugins.entries.voice-call.config.realtime.providers.openai.apiKey` for Voice
121-
Call, or the `OPENAI_API_KEY` environment variable. OpenAI OAuth profiles can
122-
still run Codex-backed `openai/*` chat models in the same install, but they do
123-
not configure Realtime voice.
123+
Call, or the `OPENAI_API_KEY` environment variable.
124124
</Note>
125125

126126
## Memory embeddings
@@ -777,7 +777,7 @@ compatibility fallback when the shared
777777
| Silence duration | `...openai.silenceDurationMs` | `500` |
778778
| Prefix padding | `...openai.prefixPaddingMs` | `300` |
779779
| Reasoning effort | `...openai.reasoningEffort` | (unset) |
780-
| Auth | `openai` API-key auth profile, `...openai.apiKey`, or `OPENAI_API_KEY` | OpenAI Platform API key required; OpenAI OAuth does not configure Realtime voice |
780+
| Auth | `openai` API-key/OAuth profile, external Codex login, `...openai.apiKey`, or `OPENAI_API_KEY` | API-key sources first; Codex OAuth fallback |
781781

782782
Available built-in Realtime voices for `gpt-realtime-2`: `alloy`, `ash`,
783783
`ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin`, `cedar`.
@@ -804,9 +804,10 @@ compatibility fallback when the shared
804804
Control UI Talk uses OpenAI browser realtime sessions with a Gateway-
805805
minted ephemeral client secret and a direct browser WebRTC SDP exchange
806806
against the OpenAI Realtime API. The Gateway mints that client secret with
807-
the selected `openai` API-key auth profile or configured OpenAI Platform
808-
API key. Gateway relay and Voice Call backend realtime WebSocket bridges
809-
use the same API-key-only auth path for native OpenAI endpoints.
807+
the selected `openai` credential. Configured keys, API-key profiles, and
808+
`OPENAI_API_KEY` take precedence; an `openai` OAuth profile or external
809+
Codex login is the fallback. Gateway relay and Voice Call backend realtime
810+
WebSocket bridges use the same credential order for native OpenAI endpoints.
810811
Maintainer live verification is available with
811812
`OPENAI_API_KEY=... GEMINI_API_KEY=... node --import tsx scripts/dev/realtime-talk-live-smoke.ts`;
812813
the OpenAI legs verify both the backend WebSocket bridge and the browser

‎docs/web/control-ui.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ The terminal is also available as a full-screen, terminal-only document at `/?vi
249249

250250
</Accordion>
251251
<Accordion title="Talk mode (browser realtime)">
252-
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key auth profile, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`; OpenAI OAuth profiles do not configure Realtime voice. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. The browser never receives a standard provider API key: OpenAI receives an ephemeral Realtime client secret for WebRTC, and Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
252+
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key/OAuth profile, an external Codex login, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`. Configured API-key sources take precedence and Codex OAuth is the automatic fallback. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. The browser never receives a standard provider API key or OAuth token: OpenAI receives an ephemeral Realtime client secret for WebRTC, and Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
253253

254254
The Chat composer includes a Talk options caret next to the Talk start/stop button. Its compact panel keeps only Voice, Model, and Sensitivity for the next Talk session. **More in Settings** opens **Settings → Communications → Talk**, where persistent provider, transport, reasoning effort, exact VAD threshold, silence duration, and prefix padding defaults live; changing those defaults requires `operator.admin` access. Blank composer values fall back to those configured defaults or the provider default. Configuring Gateway relay forces the backend relay path; configuring WebRTC keeps the session client-owned and fails instead of silently falling back to relay if the provider cannot create a browser session.
255255

‎extensions/discord/src/voice/manager.e2e.test.ts‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2252,19 +2252,24 @@ describe("DiscordVoiceManager", () => {
22522252

22532253
it("uses agent-proxy realtime voice by default", async () => {
22542254
agentCommandMock.mockResolvedValueOnce({ payloads: [{ text: "agent proxy answer" }] });
2255-
const manager = createManager({
2256-
groupPolicy: "open",
2257-
voice: {
2258-
enabled: true,
2259-
model: "openai/gpt-5.5",
2260-
realtime: {
2261-
provider: "openai",
2262-
model: "gpt-realtime-2",
2263-
speakerVoice: "cedar",
2264-
debounceMs: 1,
2255+
const cfg = { auth: { order: { openai: ["openai:codex-cli"] } } } as never;
2256+
const manager = createManager(
2257+
{
2258+
groupPolicy: "open",
2259+
voice: {
2260+
enabled: true,
2261+
model: "openai/gpt-5.5",
2262+
realtime: {
2263+
provider: "openai",
2264+
model: "gpt-realtime-2",
2265+
speakerVoice: "cedar",
2266+
debounceMs: 1,
2267+
},
22652268
},
22662269
},
2267-
});
2270+
undefined,
2271+
cfg,
2272+
);
22682273

22692274
const result = await manager.join({ guildId: "g1", channelId: "1001" });
22702275

@@ -2301,6 +2306,7 @@ describe("DiscordVoiceManager", () => {
23012306
| {
23022307
audioSink?: { sendAudio: (audio: Buffer) => void };
23032308
autoRespondToAudio?: boolean;
2309+
cfg?: unknown;
23042310
instructions?: string;
23052311
tools?: Array<{ name: string }>;
23062312
onToolCall?: (
@@ -2314,6 +2320,7 @@ describe("DiscordVoiceManager", () => {
23142320
) => void;
23152321
}
23162322
| undefined;
2323+
expect(bridgeParams?.cfg).toBe(cfg);
23172324
expect(bridgeParams?.autoRespondToAudio).toBe(false);
23182325
expect(bridgeParams?.instructions).toContain("same OpenClaw agent");
23192326
expect(bridgeParams?.instructions).toContain("short natural backchannel");

‎extensions/discord/src/voice/realtime.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -500,6 +500,7 @@ export class DiscordRealtimeVoiceSession implements VoiceRealtimeSession {
500500
});
501501
this.bridge = createRealtimeVoiceBridgeSession({
502502
provider: resolved.provider,
503+
cfg: this.params.cfg,
503504
providerConfig: resolved.providerConfig,
504505
audioFormat: REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ,
505506
instructions,

0 commit comments

Comments
 (0)