Skip to content

Commit 9566ade

Browse files
steipetesteipete
authored
fix(openai): correct Realtime auth and transcription secrets (#102518)
* fix(openai): correct Realtime auth and transcription secrets * chore: leave release notes to release automation
1 parent ea8076a commit 9566ade

8 files changed

Lines changed: 162 additions & 168 deletions

‎docs/providers/openai.md‎

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -115,22 +115,20 @@ were not set by explicit runtime config.
115115
| Text-to-speech | `messages.tts.provider: "openai"` / `tts` | Yes |
116116
| Batch speech-to-text | `tools.media.audio` / media understanding | Yes |
117117
| Streaming speech-to-text | Voice Call `streaming.provider: "openai"` | Yes |
118-
| Realtime voice | Voice Call `realtime.provider: "openai"` / Control UI Talk `talk.realtime.provider: "openai"` | Yes (OpenAI API key or Codex OAuth) |
118+
| Realtime voice | Voice Call `realtime.provider: "openai"` / Control UI Talk `talk.realtime.provider: "openai"` | Yes (OpenAI Platform API key) |
119119
| Embeddings | memory embedding provider | Yes |
120120

121121
<Note>
122122
OpenAI Realtime voice goes through the public **OpenAI Platform Realtime
123-
API**. It accepts either a Platform API key or an `openai` OAuth profile,
124-
including an automatically discovered external Codex login. API-key sessions
125-
use the key's Platform billing; OAuth availability and billing follow the
126-
authenticated account's Realtime entitlement.
123+
API** and requires a Platform API key. Codex OAuth tokens authenticate the
124+
ChatGPT Codex backend instead; they are not interchangeable with Platform API
125+
keys for the public Realtime endpoints.
127126

128127
If API-key auth reports missing billing, top up Platform credits at
129128
[platform.openai.com/account/billing](https://platform.openai.com/account/billing)
130129
for the organization backing your realtime credentials when using API-key
131130
auth. Realtime voice accepts the `openai` API-key auth profile created by
132-
`openclaw onboard --auth-choice openai-api-key`, an `openai` OAuth profile or
133-
external Codex login, a Platform `OPENAI_API_KEY` set via
131+
`openclaw onboard --auth-choice openai-api-key`, a Platform API key set via
134132
`talk.realtime.providers.openai.apiKey` for Control UI Talk, or
135133
`plugins.entries.voice-call.config.realtime.providers.openai.apiKey` for Voice
136134
Call, or the `OPENAI_API_KEY` environment variable.
@@ -763,12 +761,12 @@ compatibility fallback when the shared
763761
| Prompt | `...openai.prompt` | (unset) |
764762
| Silence duration | `...openai.silenceDurationMs` | `800` |
765763
| VAD threshold | `...openai.vadThreshold` | `0.5` |
766-
| Auth | `...openai.apiKey`, `OPENAI_API_KEY`, or `openai` OAuth | API keys connect directly; OAuth mints a Realtime transcription client secret |
764+
| Auth | `...openai.apiKey`, `OPENAI_API_KEY`, or `openai` API-key profile | Platform API key required |
767765

768766
<Note>
769767
Uses a WebSocket connection to `wss://api.openai.com/v1/realtime` with
770-
G.711 u-law (`g711_ulaw` / `audio/pcmu`) audio. When only `openai` OAuth is
771-
configured, the Gateway mints an ephemeral Realtime transcription client
768+
G.711 u-law (`g711_ulaw` / `audio/pcmu`) audio. For an `openai` API-key
769+
profile, the Gateway mints an ephemeral Realtime transcription client
772770
secret before opening the WebSocket. This streaming provider is for Voice
773771
Call's realtime transcription path; Discord voice currently records short
774772
segments and uses the batch `tools.media.audio` transcription path
@@ -790,7 +788,7 @@ compatibility fallback when the shared
790788
| Silence duration | `...openai.silenceDurationMs` | `500` |
791789
| Prefix padding | `...openai.prefixPaddingMs` | `300` |
792790
| Reasoning effort | `...openai.reasoningEffort` | (unset) |
793-
| Auth | `openai` API-key/OAuth profile, external Codex login, `...openai.apiKey`, or `OPENAI_API_KEY` | API-key sources first; Codex OAuth fallback |
791+
| Auth | `openai` API-key profile, `...openai.apiKey`, or `OPENAI_API_KEY` | OpenAI Platform API key required |
794792

795793
Available built-in Realtime voices for `gpt-realtime-2.1`: `alloy`, `ash`,
796794
`ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin`, `cedar`.

‎docs/web/control-ui.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ The terminal is also available as a full-screen, terminal-only document at `/?vi
257257

258258
</Accordion>
259259
<Accordion title="Talk mode (browser realtime)">
260-
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key/OAuth profile, an external Codex login, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`. Configured API-key sources take precedence and Codex OAuth is the automatic fallback. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. The browser never receives a standard provider API key or OAuth token: OpenAI receives an ephemeral Realtime client secret for WebRTC, and Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
260+
Talk mode uses a registered realtime voice provider. Configure OpenAI with `talk.realtime.provider: "openai"` plus an `openai` API-key profile, `talk.realtime.providers.openai.apiKey`, or `OPENAI_API_KEY`. OpenAI Realtime uses the public Platform API and requires a Platform API key; a Codex OAuth login does not satisfy this surface. Configure Google with `talk.realtime.provider: "google"` plus `talk.realtime.providers.google.apiKey`. The browser never receives a standard provider API key: OpenAI receives an ephemeral Realtime client secret for WebRTC, and Google Live receives a one-use constrained Live API auth token for a browser WebSocket session, with instructions and tool declarations locked into the token by the Gateway. Providers that only expose a backend realtime bridge run through the Gateway relay transport, so credentials and vendor sockets stay server-side while browser audio moves through authenticated Gateway RPCs. The Realtime session prompt is assembled by the Gateway; `talk.client.create` does not accept caller-provided instruction overrides.
261261

262262
Persistent provider, model, voice, transport, reasoning effort, exact VAD threshold, silence duration, and prefix padding defaults live in **Settings → Communications → Talk**; changing them requires `operator.admin` access. Configuring Gateway relay forces the backend relay path; configuring WebRTC keeps the session client-owned and fails instead of silently falling back to relay if the provider cannot create a browser session.
263263

‎extensions/openai/realtime-provider-shared.test.ts‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,44 @@ describe("createOpenAIRealtimeClientSecret", () => {
9797

9898
expect(streamed.wasCanceled()).toBe(true);
9999
});
100+
101+
it("creates transcription secrets through the current client-secrets endpoint", async () => {
102+
guardedFetch(
103+
new Response(JSON.stringify({ value: "ek-transcription", expires_at: 1_800_000_000 }), {
104+
status: 200,
105+
}),
106+
);
107+
108+
await createOpenAIRealtimeTranscriptionClientSecret({
109+
authToken: "sk-test",
110+
auditContext: "test",
111+
session: { type: "transcription" },
112+
});
113+
114+
expect(fetchWithSsrFGuardMock).toHaveBeenCalledWith(
115+
expect.objectContaining({
116+
url: "https://api.openai.com/v1/realtime/client_secrets",
117+
init: expect.objectContaining({
118+
body: JSON.stringify({ session: { type: "transcription" } }),
119+
}),
120+
}),
121+
);
122+
});
123+
124+
it("replaces rejected transcription API-key details with bounded guidance", async () => {
125+
guardedFetch(
126+
new Response(JSON.stringify({ error: { message: "Incorrect API key provided: secret" } }), {
127+
status: 401,
128+
}),
129+
);
130+
131+
await expect(
132+
createOpenAIRealtimeTranscriptionClientSecret({
133+
authToken: "sk-test",
134+
auditContext: "test",
135+
session: { type: "transcription" },
136+
authRejectedMessage: "Update the transcription API key",
137+
}),
138+
).rejects.toThrow("Update the transcription API key");
139+
});
100140
});

‎extensions/openai/realtime-provider-shared.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -171,12 +171,13 @@ export async function createOpenAIRealtimeTranscriptionClientSecret(params: {
171171
authToken: string;
172172
auditContext: string;
173173
session: Record<string, unknown>;
174+
authRejectedMessage?: string;
174175
}): Promise<OpenAIRealtimeClientSecretResult> {
175-
const url = `${OPENAI_REALTIME_API_BASE_URL}/realtime/transcription_sessions`;
176+
const url = `${OPENAI_REALTIME_API_BASE_URL}/realtime/client_secrets`;
176177
return createOpenAIRealtimeSecret({
177178
...params,
178179
url,
179-
body: params.session,
180+
body: { session: params.session },
180181
errorMessage: "OpenAI Realtime transcription client secret failed",
181182
missingValueMessage:
182183
"OpenAI Realtime transcription client secret response did not include a value",

‎extensions/openai/realtime-transcription-provider.test.ts‎

Lines changed: 61 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -203,7 +203,7 @@ describe("buildOpenAIRealtimeTranscriptionProvider", () => {
203203
expect(provider.aliases).toContain("openai-realtime");
204204
});
205205

206-
it("treats a Codex OAuth profile as configured when no API key is present", () => {
206+
it("treats an OpenAI API-key profile as configured", () => {
207207
const provider = buildOpenAIRealtimeTranscriptionProvider();
208208
const cfg = { auth: { order: { openai: ["openai:default"] } } };
209209
providerAuthMocks.isProviderAuthProfileConfigured.mockReturnValue(true);
@@ -212,13 +212,14 @@ describe("buildOpenAIRealtimeTranscriptionProvider", () => {
212212
expect(providerAuthMocks.isProviderAuthProfileConfigured).toHaveBeenCalledWith({
213213
provider: "openai",
214214
cfg,
215+
profileTypes: ["api_key"],
215216
});
216217
});
217218

218-
it("mints a Codex OAuth client secret for realtime transcription sockets", async () => {
219+
it("mints an API-key client secret for realtime transcription sockets", async () => {
219220
const provider = buildOpenAIRealtimeTranscriptionProvider();
220221
const release = vi.fn();
221-
providerAuthMocks.resolveProviderAuthProfileApiKey.mockResolvedValue("oauth-token");
222+
providerAuthMocks.resolveProviderAuthProfileApiKey.mockResolvedValue("sk-profile"); // pragma: allowlist secret
222223
ssrfMocks.fetchWithSsrFGuard.mockResolvedValue({
223224
response: new Response(JSON.stringify({ value: "ek-test" }), { status: 200 }),
224225
release,
@@ -236,10 +237,11 @@ describe("buildOpenAIRealtimeTranscriptionProvider", () => {
236237
expect(providerAuthMocks.resolveProviderAuthProfileApiKey).toHaveBeenCalledWith({
237238
provider: "openai",
238239
cfg,
240+
profileTypes: ["api_key"],
239241
});
240242
const request = mockCallArg(ssrfMocks.fetchWithSsrFGuard);
241243
expect(request.auditContext).toBe("openai-realtime-transcription-session");
242-
expect(request.url).toBe("https://api.openai.com/v1/realtime/transcription_sessions");
244+
expect(request.url).toBe("https://api.openai.com/v1/realtime/client_secrets");
243245
expect(request.policy).toEqual({
244246
allowRfc2544BenchmarkRange: true,
245247
allowIpv6UniqueLocalRange: true,
@@ -251,20 +253,22 @@ describe("buildOpenAIRealtimeTranscriptionProvider", () => {
251253
body?: unknown;
252254
};
253255
expect(init.method).toBe("POST");
254-
expect(init.headers?.Authorization).toBe("Bearer oauth-token");
256+
expect(init.headers?.Authorization).toBe("Bearer sk-profile");
255257
expect(init.headers?.["Content-Type"]).toBe("application/json");
256258
expect(typeof init.body).toBe("string");
257259
expect(JSON.parse(init.body as string)).toEqual({
258-
type: "transcription",
259-
audio: {
260-
input: {
261-
format: { type: "audio/pcmu" },
262-
transcription: { model: "gpt-4o-transcribe" },
263-
turn_detection: {
264-
type: "server_vad",
265-
threshold: 0.5,
266-
prefix_padding_ms: 300,
267-
silence_duration_ms: 800,
260+
session: {
261+
type: "transcription",
262+
audio: {
263+
input: {
264+
format: { type: "audio/pcmu" },
265+
transcription: { model: "gpt-4o-transcribe" },
266+
turn_detection: {
267+
type: "server_vad",
268+
threshold: 0.5,
269+
prefix_padding_ms: 300,
270+
silence_duration_ms: 800,
271+
},
268272
},
269273
},
270274
},
@@ -297,6 +301,48 @@ describe("buildOpenAIRealtimeTranscriptionProvider", () => {
297301
session.close();
298302
});
299303

304+
it("does not use Codex OAuth for realtime transcription", async () => {
305+
const provider = buildOpenAIRealtimeTranscriptionProvider();
306+
const cfg = { auth: { order: { openai: ["openai:default"] } } };
307+
const session = provider.createSession({ cfg: cfg as never, providerConfig: {} });
308+
309+
await expect(session.connect()).rejects.toThrow(
310+
"OpenAI Realtime transcription requires an OpenAI Platform API key",
311+
);
312+
expect(providerAuthMocks.resolveProviderAuthProfileApiKey).toHaveBeenCalledWith({
313+
provider: "openai",
314+
cfg,
315+
profileTypes: ["api_key"],
316+
});
317+
expect(ssrfMocks.fetchWithSsrFGuard).not.toHaveBeenCalled();
318+
expect(FakeWebSocket.instances).toHaveLength(0);
319+
});
320+
321+
it("prefers an API-key profile over OPENAI_API_KEY", async () => {
322+
vi.stubEnv("OPENAI_API_KEY", "sk-env"); // pragma: allowlist secret
323+
providerAuthMocks.resolveProviderAuthProfileApiKey.mockResolvedValue("sk-profile"); // pragma: allowlist secret
324+
ssrfMocks.fetchWithSsrFGuard.mockResolvedValue({
325+
response: new Response(JSON.stringify({ value: "ek-test" }), { status: 200 }),
326+
release: vi.fn(),
327+
});
328+
const provider = buildOpenAIRealtimeTranscriptionProvider();
329+
const session = provider.createSession({ providerConfig: {} });
330+
331+
const connecting = session.connect();
332+
const socket = await waitForFakeSocket();
333+
334+
expect(socket.headers?.Authorization).toBe("Bearer ek-test");
335+
const request = mockCallArg(ssrfMocks.fetchWithSsrFGuard);
336+
const init = request.init as { headers?: Record<string, string> };
337+
expect(init.headers?.Authorization).toBe("Bearer sk-profile");
338+
339+
socket.readyState = FakeWebSocket.OPEN;
340+
socket.emit("open");
341+
socket.emit("message", Buffer.from(JSON.stringify({ type: "transcription_session.updated" })));
342+
await connecting;
343+
session.close();
344+
});
345+
300346
it("waits for the OpenAI session update before draining audio", async () => {
301347
const provider = buildOpenAIRealtimeTranscriptionProvider();
302348
const session = provider.createSession({

‎extensions/openai/realtime-transcription-provider.ts‎

Lines changed: 21 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,10 @@ const OPENAI_REALTIME_TRANSCRIPTION_CONNECT_TIMEOUT_MS = 10_000;
7373
const OPENAI_REALTIME_TRANSCRIPTION_MAX_RECONNECT_ATTEMPTS = 5;
7474
const OPENAI_REALTIME_TRANSCRIPTION_RECONNECT_DELAY_MS = 1000;
7575
const OPENAI_REALTIME_TRANSCRIPTION_DEFAULT_MODEL = "gpt-4o-transcribe";
76+
const OPENAI_REALTIME_TRANSCRIPTION_API_KEY_REQUIRED =
77+
"OpenAI Realtime transcription requires an OpenAI Platform API key";
78+
const OPENAI_REALTIME_TRANSCRIPTION_API_KEY_REJECTED =
79+
"OpenAI Realtime transcription rejected the selected API key. Update or remove the active OpenAI API-key source";
7680

7781
function normalizeProviderConfig(
7882
config: RealtimeTranscriptionProviderConfig,
@@ -139,23 +143,28 @@ function buildOpenAIRealtimeTranscriptionSessionPayload(
139143
async function resolveOpenAIRealtimeTranscriptionAuthorization(
140144
config: OpenAIRealtimeTranscriptionSessionConfig,
141145
): Promise<string> {
142-
const apiKey = config.apiKey || process.env.OPENAI_API_KEY;
143-
if (apiKey) {
144-
return apiKey;
146+
if (config.apiKey) {
147+
return config.apiKey;
145148
}
146149
const authToken = await resolveProviderAuthProfileApiKey({
147150
provider: "openai",
148151
cfg: config.cfg,
152+
profileTypes: ["api_key"],
149153
});
150-
if (!authToken) {
151-
throw new Error("OpenAI API key or Codex OAuth missing");
154+
if (authToken) {
155+
const clientSecret = await createOpenAIRealtimeTranscriptionClientSecret({
156+
authToken,
157+
auditContext: "openai-realtime-transcription-session",
158+
session: buildOpenAIRealtimeTranscriptionSessionPayload(config),
159+
authRejectedMessage: OPENAI_REALTIME_TRANSCRIPTION_API_KEY_REJECTED,
160+
});
161+
return clientSecret.value;
152162
}
153-
const clientSecret = await createOpenAIRealtimeTranscriptionClientSecret({
154-
authToken,
155-
auditContext: "openai-realtime-transcription-session",
156-
session: buildOpenAIRealtimeTranscriptionSessionPayload(config),
157-
});
158-
return clientSecret.value;
163+
const envApiKey = process.env.OPENAI_API_KEY?.trim();
164+
if (envApiKey) {
165+
return envApiKey;
166+
}
167+
throw new Error(OPENAI_REALTIME_TRANSCRIPTION_API_KEY_REQUIRED);
159168
}
160169

161170
function createOpenAIRealtimeTranscriptionSession(
@@ -260,7 +269,7 @@ export function buildOpenAIRealtimeTranscriptionProvider(): RealtimeTranscriptio
260269
Boolean(
261270
normalizeProviderConfig(providerConfig).apiKey ||
262271
process.env.OPENAI_API_KEY ||
263-
isProviderAuthProfileConfigured({ provider: "openai", cfg }),
272+
isProviderAuthProfileConfigured({ provider: "openai", cfg, profileTypes: ["api_key"] }),
264273
),
265274
createSession: (req) => {
266275
const config = normalizeProviderConfig(req.providerConfig);

0 commit comments

Comments
 (0)