Skip to content

Commit 9f953d8

Browse files
committed
Merge PR #379 (fix/proxy-v1-live) into dev: GPT-Live /v1/live voice relay + sideband WebSocket
Relay ChatGPT/Codex App voice (GPT-Live / Frameless Bidi) call-create and the Frameless/Realtime sideband WebSocket so voice works under Design B injection (#371). Verified green on top of dev: typecheck + live/auth tests + contract match vs codex-cli.
2 parents 3d83dd0 + 5e55fe5 commit 9f953d8

11 files changed

Lines changed: 1165 additions & 6 deletions

File tree

‎docs-site/src/content/docs/ja/reference/architecture.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ HTTP の境界は `server/index.ts` が担い、Responses データプレーン
4545
1. `server/index.ts` で CORS と API 認証を確認し、終了待ち状態なら新規リクエストを拒否したのち、リクエストのライフサイクルを記録します。ここで `GET /v1/models`、`POST /v1/responses`、
4646
`POST /v1/responses/compact`、`POST /v1/images/generations` / `POST /v1/images/edits`
4747
(Codex 組み込み `image_gen` ツール用 — `server/images.ts` が OpenAI 系の上流に中継)、
48+
`POST /v1/live` / `POST /v1/realtime/calls`(ChatGPT / Codex App 音声と OpenAI Realtime
49+
の call-create、`server/live.ts` が中継)と `/v1/live/{callId}` サイドバンド WebSocket、
4850
`/v1/responses` のオプション WebSocket アップグレードを提供します。
4951
2. `server/responses/core.ts` が展開し JSON を読みます。覚えておいた `previous_response_id` 入力があれば展開したのち `responses/parser.ts` に渡します。
5052
3. `router.ts` が通常のモデル id または `provider/model` id を解決します。続いて Codex アカウント affinity を決定し、必要ならプロバイダー OAuth を更新して選択された認証情報を route に適用します。

‎docs-site/src/content/docs/ko/reference/architecture.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ HTTP 경계는 `server/index.ts`가 맡고, Responses 데이터 플레인은 `se
4747
주기를 기록합니다. 여기서 `GET /v1/models`, `POST /v1/responses`,
4848
`POST /v1/responses/compact`, `POST /v1/images/generations` / `POST /v1/images/edits`
4949
(Codex 내장 `image_gen` 도구용 — `server/images.ts`가 OpenAI 계열 업스트림으로 중계),
50-
`/v1/responses`의 선택적 WebSocket 업그레이드를 제공합니다.
50+
`POST /v1/live` / `POST /v1/realtime/calls`(ChatGPT / Codex App 음성 및 OpenAI Realtime
51+
호출 생성, `server/live.ts`가 중계)와 `/v1/live/{callId}` 사이드밴드 WebSocket,
52+
그리고 `/v1/responses`의 선택적 WebSocket 업그레이드를 제공합니다.
5153
2. `server/responses/core.ts`가 압축을 풀고 JSON을 읽습니다. 기억해 둔 `previous_response_id` 입력이 있으면
5254
펼친 다음 `responses/parser.ts`로 넘깁니다.
5355
3. `router.ts`가 일반 모델 id 또는 `provider/model` id를 해석합니다. 이어서 Codex 계정 affinity를

‎docs-site/src/content/docs/reference/architecture.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,10 @@ the `server/responses.ts` facade and its `server/responses/*.ts` modules:
4848
records request lifecycle metadata. It serves `GET /v1/models`, `POST /v1/responses`,
4949
`POST /v1/responses/compact`, `POST /v1/images/generations` / `POST /v1/images/edits`
5050
(relayed to an OpenAI-family upstream by `server/images.ts` for codex's built-in `image_gen`
51-
tool), and the optional WebSocket upgrade on `/v1/responses`.
51+
tool), `POST /v1/live` / `POST /v1/realtime/calls` (ChatGPT / Codex App voice and OpenAI
52+
Realtime call-create, relayed by `server/live.ts`), sideband WebSocket joins on
53+
`/v1/live/{callId}` (and `/v1/realtime?call_id=`), and the optional WebSocket upgrade on
54+
`/v1/responses`.
5255
2. `server/responses/core.ts` decompresses and parses JSON, expands locally remembered
5356
`previous_response_id` input when available, then calls `responses/parser.ts`.
5457
3. `router.ts` resolves a bare or `provider/model` id. The server then resolves Codex account

‎docs-site/src/content/docs/ru/reference/architecture.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,9 @@ src/
5050
`GET /v1/models`, `POST /v1/responses`,
5151
`POST /v1/responses/compact`, `POST /v1/images/generations` / `POST /v1/images/edits`
5252
(ретранслируются модулем `server/images.ts` вышестоящему провайдеру семейства OpenAI для
53-
встроенного инструмента codex `image_gen`), а также необязательный WebSocket-апгрейд на
53+
встроенного инструмента codex `image_gen`), `POST /v1/live` / `POST /v1/realtime/calls`
54+
(создание голосового/Realtime-вызова ChatGPT / Codex App, ретранслируется `server/live.ts`),
55+
sideband WebSocket на `/v1/live/{callId}`, а также необязательный WebSocket-апгрейд на
5456
`/v1/responses`.
5557
2. `server/responses/core.ts` распаковывает и парсит JSON, разворачивает локально запомненный вход
5658
`previous_response_id`, когда он доступен, затем вызывает `responses/parser.ts`.

‎docs-site/src/content/docs/zh-cn/reference/architecture.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ src/
4747
1. `server/index.ts` 应用 CORS 和 API 认证,在 drain 期间拒绝新请求,并记录请求生命周期
4848
metadata。它提供 `GET /v1/models`、`POST /v1/responses`、
4949
`POST /v1/responses/compact`、`POST /v1/images/generations` / `POST /v1/images/edits`
50-
(供 Codex 内置 `image_gen` 工具使用——由 `server/images.ts` 中继到 OpenAI 系上游),
50+
(供 Codex 内置 `image_gen` 工具使用——由 `server/images.ts` 中继到 OpenAI 系上游)、
51+
`POST /v1/live` / `POST /v1/realtime/calls`(ChatGPT / Codex App 语音与 OpenAI Realtime
52+
建连,由 `server/live.ts` 中继)、`/v1/live/{callId}` 旁路 WebSocket,
5153
以及 `/v1/responses` 上可选的 WebSocket upgrade。
5254
2. `server/responses/core.ts` 解压并解析 JSON;如果本地记住了对应输入,则展开
5355
`previous_response_id`,随后调用 `responses/parser.ts`。

‎src/server/auth-cors.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,9 @@ export function corsHeaders(req?: Request, config?: OcxConfig): Record<string, s
8080
return {
8181
"Access-Control-Allow-Origin": allowOrigin,
8282
"Access-Control-Allow-Methods": "GET, POST, PUT, PATCH, DELETE, OPTIONS",
83-
"Access-Control-Allow-Headers": "Content-Type, Authorization, X-OpenCodex-API-Key, X-Api-Key, Anthropic-Version, Anthropic-Beta",
83+
// ChatGPT-Account-Id is required for browser/Electron ChatGPT & Codex App voice preflights
84+
// (direct forward auth matches the bearer to this account id).
85+
"Access-Control-Allow-Headers": "Content-Type, Authorization, X-OpenCodex-API-Key, X-Api-Key, Anthropic-Version, Anthropic-Beta, ChatGPT-Account-Id",
8486
"Vary": "Origin",
8587
};
8688
}

‎src/server/index.ts‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,11 +122,90 @@ import { handleChatCompletions } from "./chat-completions";
122122
import { anthropicErrorResponse } from "../claude/outbound";
123123
import { buildDesktop3pRegistry } from "../claude/desktop-3p";
124124
import { handleImages } from "./images";
125+
import { handleLive, parseLiveSidebandTarget, resolveLiveSidebandUpgrade } from "./live";
125126
import { handleSearch } from "./search";
126127
import { fetchAllModels, handleManagementAPI, VERSION } from "./management-api";
127128

128129
const MAX_WS_FRAME_BYTES = 50 * 1024 * 1024;
129130
const WEBSOCKET_IDLE_TIMEOUT_SECONDS = 0;
131+
const LIVE_SIDEBAND_PENDING_MAX = 32;
132+
133+
function closeLiveSideband(ws: ServerWebSocket<WsData>, code = 1000, reason = ""): void {
134+
try {
135+
ws.data.liveUpstream?.close(code, reason);
136+
} catch {
137+
/* upstream already gone */
138+
}
139+
ws.data.liveUpstream = undefined;
140+
ws.data.livePending = undefined;
141+
try {
142+
if (ws.readyState === WebSocket.OPEN || ws.readyState === WebSocket.CONNECTING) {
143+
ws.close(code, reason);
144+
}
145+
} catch {
146+
/* client already gone */
147+
}
148+
}
149+
150+
function attachLiveSidebandUpstream(ws: ServerWebSocket<WsData>): void {
151+
const url = ws.data.liveUpstreamUrl;
152+
if (!url) {
153+
closeLiveSideband(ws, 1011, "missing upstream");
154+
return;
155+
}
156+
let upstream: WebSocket;
157+
try {
158+
// Bun accepts per-handshake headers; the DOM lib types only list protocol arrays.
159+
upstream = new WebSocket(url, { headers: ws.data.liveUpstreamHeaders ?? {} } as unknown as string[]);
160+
} catch {
161+
closeLiveSideband(ws, 1011, "upstream connect failed");
162+
return;
163+
}
164+
ws.data.liveUpstream = upstream;
165+
ws.data.cancel = () => {
166+
try {
167+
upstream.close(1000, "client closed");
168+
} catch {
169+
/* ignore */
170+
}
171+
};
172+
173+
upstream.addEventListener("open", () => {
174+
ws.data.liveOpened = true;
175+
const pending = ws.data.livePending ?? [];
176+
ws.data.livePending = undefined;
177+
for (const frame of pending) {
178+
try {
179+
upstream.send(frame);
180+
} catch {
181+
closeLiveSideband(ws, 1011, "upstream send failed");
182+
return;
183+
}
184+
}
185+
});
186+
upstream.addEventListener("message", (event) => {
187+
try {
188+
if (typeof event.data === "string") ws.send(event.data);
189+
else if (event.data instanceof ArrayBuffer) ws.send(event.data);
190+
else if (ArrayBuffer.isView(event.data)) {
191+
ws.send(event.data.buffer.slice(event.data.byteOffset, event.data.byteOffset + event.data.byteLength));
192+
} else ws.send(event.data as Buffer);
193+
} catch {
194+
closeLiveSideband(ws, 1011, "client send failed");
195+
}
196+
});
197+
upstream.addEventListener("close", (event) => {
198+
try {
199+
ws.close(event.code || 1000, event.reason || "");
200+
} catch {
201+
/* ignore */
202+
}
203+
ws.data.liveUpstream = undefined;
204+
});
205+
upstream.addEventListener("error", () => {
206+
closeLiveSideband(ws, 1011, "upstream error");
207+
});
208+
}
130209

131210
// GUI static serving extracted to ./server/gui-static. Re-exported below to keep the
132211
// "../src/server" import surface stable for tests/callers.
@@ -498,6 +577,77 @@ export function startServer(port?: number) {
498577
return withCors(response, req, config);
499578
}
500579

580+
// ChatGPT / Codex App voice (GPT‑Live / Frameless Bidi) + OpenAI Realtime call-create.
581+
// Clients hit either /v1/live (Frameless App) or /v1/realtime/calls (codex RealtimeCallClient /
582+
// public Realtime API). Sideband WS joins are handled just below.
583+
if (
584+
req.method === "POST"
585+
&& (url.pathname === "/v1/live" || url.pathname === "/v1/realtime/calls")
586+
) {
587+
disableResponsesRequestTimeout(req, requestServer);
588+
if (isDraining()) {
589+
return new Response("Service shutting down", {
590+
status: 503,
591+
headers: { ...corsHeaders(req, config), "Retry-After": "5" },
592+
});
593+
}
594+
const apiAuthError = requireApiAuth(req, config, "data-plane");
595+
if (apiAuthError) return withCors(apiAuthError, req, config);
596+
if (!isAllowedRequestOrigin(req, config)) {
597+
return withCors(formatErrorResponse(403, "origin_rejected", "cross-origin data-plane request blocked"), req, config);
598+
}
599+
const start = Date.now();
600+
const requestId = nextRequestLogId(start);
601+
const logCtx: RequestLogContext = { model: "gpt-live", provider: "unknown" };
602+
const response = await handleLive(req, config, logCtx);
603+
addFinalRequestLog(
604+
requestId,
605+
start,
606+
logCtx,
607+
response.status,
608+
response.status === 499 ? { closeReason: "client_cancel" } : undefined,
609+
);
610+
return withCors(response, req, config);
611+
}
612+
613+
// Voice / Realtime sideband WebSocket: Frameless joins /v1/live/{callId}; Realtime v1 joins
614+
// /v1/realtime?call_id= (or /v1/realtime/calls/{callId}). Transparent bidirectional relay.
615+
const liveSidebandTarget = req.headers.get("upgrade")?.toLowerCase() === "websocket"
616+
? parseLiveSidebandTarget(url.pathname, url.searchParams)
617+
: null;
618+
if (liveSidebandTarget) {
619+
if (isDraining()) {
620+
return new Response("Service shutting down", {
621+
status: 503,
622+
headers: { ...corsHeaders(req, config), "Retry-After": "5" },
623+
});
624+
}
625+
const apiAuthError = requireApiAuth(req, config, "data-plane");
626+
if (apiAuthError) return withCors(apiAuthError, req, config);
627+
if (!isAllowedRequestOrigin(req, config)) {
628+
return withCors(formatErrorResponse(403, "origin_rejected", "WebSocket upgrade blocked: non-local Origin"), req, config);
629+
}
630+
const start = Date.now();
631+
const requestId = nextRequestLogId(start);
632+
const logCtx: RequestLogContext = { model: "gpt-live", provider: "unknown" };
633+
const resolved = await resolveLiveSidebandUpgrade(req, config, logCtx, liveSidebandTarget);
634+
if (resolved instanceof Response) {
635+
addFinalRequestLog(requestId, start, logCtx, resolved.status);
636+
return withCors(resolved, req, config);
637+
}
638+
addFinalRequestLog(requestId, start, logCtx, 101);
639+
if (server.upgrade(req, {
640+
data: {
641+
kind: "live-sideband",
642+
liveUpstreamUrl: resolved.upstreamWsUrl,
643+
liveUpstreamHeaders: resolved.headers,
644+
livePending: [],
645+
liveOpened: false,
646+
} satisfies WsData,
647+
})) return undefined as unknown as Response;
648+
return withCors(formatErrorResponse(426, "upgrade_required", "WebSocket upgrade failed"), req, config);
649+
}
650+
501651
// Data-plane guard: unknown /v1/* paths must fail with JSON 404, never fall through to the
502652
// GUI static handler (extensionless paths would get index.html with HTTP 200 and codex-rs
503653
// endpoint clients — memories/*, realtime/* — would surface confusing
@@ -519,10 +669,37 @@ export function startServer(port?: number) {
519669
// Responses WebSocket data plane (phase 120.2). Re-frames the same SSE pipeline onto the
520670
// socket: parse response.create → run handleResponses unchanged → pump its SSE body as WS
521671
// Text frames. response.processed is a no-op ack. close() aborts the upstream (RC2 parity).
672+
// Live sideband sockets (kind=live-sideband) are a transparent bidirectional relay instead.
522673
open(ws: ServerWebSocket<WsData>) {
674+
if (ws.data.kind === "live-sideband") {
675+
attachLiveSidebandUpstream(ws);
676+
return;
677+
}
523678
registerCodexWebSocket(ws);
524679
},
525680
message(ws: ServerWebSocket<WsData>, raw: string | Buffer) {
681+
if (ws.data.kind === "live-sideband") {
682+
const upstream = ws.data.liveUpstream;
683+
if (!upstream || upstream.readyState === WebSocket.CONNECTING || !ws.data.liveOpened) {
684+
const pending = ws.data.livePending ?? (ws.data.livePending = []);
685+
if (pending.length >= LIVE_SIDEBAND_PENDING_MAX) {
686+
closeLiveSideband(ws, 1009, "too many pending frames");
687+
return;
688+
}
689+
pending.push(raw);
690+
return;
691+
}
692+
if (upstream.readyState !== WebSocket.OPEN) {
693+
closeLiveSideband(ws, 1011, "upstream not open");
694+
return;
695+
}
696+
try {
697+
upstream.send(raw);
698+
} catch {
699+
closeLiveSideband(ws, 1011, "upstream send failed");
700+
}
701+
return;
702+
}
526703
const rawBytes = typeof raw === "string" ? Buffer.byteLength(raw) : raw.byteLength;
527704
if (rawBytes > MAX_WS_FRAME_BYTES) {
528705
sendJsonFrame(ws, buildWsErrorFrame(413, {
@@ -639,6 +816,11 @@ export function startServer(port?: number) {
639816
})();
640817
},
641818
close(ws: ServerWebSocket<WsData>) {
819+
if (ws.data.kind === "live-sideband") {
820+
ws.data.cancel?.();
821+
ws.data.liveUpstream = undefined;
822+
return;
823+
}
642824
unregisterCodexWebSocket(ws);
643825
ws.data.cancel?.(); // RC2: abort the upstream when the client disconnects
644826
},

0 commit comments

Comments
 (0)