Skip to content

Commit 7d9ad2f

Browse files
axisrowclaude
andauthored
fix(app-server): bound request() so a wedged spawned app-server can't hang the client (closes #31) (#32)
* fix(app-server): bound request() so a wedged spawned app-server can't hang the client (closes #31) SpawnedCodexAppServerClient.initialize had the same unbounded handshake shape PR #30 fixed only on the broker path: request("initialize") had no deadline, so a spawned codex app-server that accepted stdin but never answered hung the client forever. RED-first: a stalled-initialize fake-codex behavior reproduces the hang (connect() never settles within 2s). Deeper fix (as promised in #31 / upstream openai#509), not a third copy of the bounded-handshake pattern: - AppServerClientBase.request(method, params, options) gains an optional timeoutMs (+ onTimeout for transport cleanup). Optional/undefined-default, so all existing 2-arg call sites are unchanged. Timer is unref'd, cleared in handleLine/handleExit. Timeout rejects with BROKER_HANDSHAKE_TIMEOUT_CODE and removes the pending entry; a late answer for that id is a no-op in handleLine. - SpawnedCodexAppServerClient.initialize now passes timeoutMs (10s default, spawnedInitializeTimeoutMs override), and kills the child on handshake failure so a timeout doesn't orphan the spawned process. - BrokerCodexAppServerClient.initialize drops its inline Promise.race and uses request(timeoutMs, onTimeout=socket.destroy) — one mechanism, both transports. - CodexAppServerClient.connect tags broker-transport init failures with brokerFatal; withAppServer's fallback OR-chain collapses to (broker-busy) || (brokerFatal). Verified: RED→GREEN (304ms, EBROKERTIMEOUT); broker suite 11/11 green; npm test unchanged (4 pre-existing failures). Co-Authored-By: Claude <[email protected]> * fix(broker): restore post-handshake broker/busy direct fallback (cycle-review critical) Codex adversarial review caught that collapsing withAppServer's fallback onto error.transport broke the #29 path: broker/busy (-32001) arrives AFTER a successful handshake, while the error is created by createProtocolError with rpcCode but NO transport (transport is tagged only on handshake failure in connect()'s catch). So every overlapping /codex:* command failed for the duration of a long-running broker turn instead of falling back to direct. Restore client?.transport === "broker" for the busy branch (post-handshake, client is assigned); keep error.brokerFatal for handshake failures (client is null, tagged in connect). Adds spawnBusyBroker helper + an end-to-end test that drives a successful initialize, then -32001 on the first RPC, and verifies the operation completes via the direct fallback. Co-Authored-By: Claude <[email protected]> * fix(app-server): make timeout-error code assignments type-safe (cycle-review critical) Codex adversarial review caught that `error.code = REQUEST_TIMEOUT_CODE` on a plain `new Error(...)` fails the repo's mandatory `npm run build` (TS2339: Property 'code' does not exist on type 'Error'). CI runs this build, so the head could not pass its shipping gate. The same pre-existing breakage in BrokerCodexAppServerClient's handshakeError (landed in #30) is fixed too. Cast the errors to ProtocolError (extended to include `code?: string`), the same pattern createProtocolError already uses. Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: axisrow <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent b057f64 commit 7d9ad2f

6 files changed

Lines changed: 264 additions & 53 deletions

File tree

‎plugins/codex/scripts/lib/app-server.mjs‎

Lines changed: 89 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* @typedef {Error & { data?: unknown, rpcCode?: number }} ProtocolError
2+
* @typedef {Error & { code?: string, data?: unknown, rpcCode?: number }} ProtocolError
33
* @typedef {import("./app-server-protocol").AppServerMethod} AppServerMethod
44
* @typedef {import("./app-server-protocol").AppServerNotification} AppServerNotification
55
* @typedef {import("./app-server-protocol").AppServerNotificationHandler} AppServerNotificationHandler
@@ -22,13 +22,16 @@ const PLUGIN_MANIFEST = JSON.parse(fs.readFileSync(PLUGIN_MANIFEST_URL, "utf8"))
2222
export const BROKER_ENDPOINT_ENV = "CODEX_COMPANION_APP_SERVER_ENDPOINT";
2323
export const BROKER_BUSY_RPC_CODE = -32001;
2424

25-
// Error code attached to timeouts during the broker handshake, so withAppServer
26-
// can recognize them and fall back to a direct (non-broker) app-server. A
27-
// wedged broker — connect accepted but initialize never answered — must not
28-
// hang a second /codex:* command forever (fork #29 / upstream openai#509).
29-
export const BROKER_HANDSHAKE_TIMEOUT_CODE = "EBROKERTIMEOUT";
25+
// Error code attached to app-server request/handshake timeouts, so withAppServer
26+
// can recognize them and fall back to a direct (non-broker) app-server. A wedged
27+
// app-server — connect accepted but a request (initialize, ...) never answered —
28+
// must not hang a /codex:* command forever (fork #29/#31 / upstream openai#509).
29+
// Despite the "EBROKER" prefix the code is transport-agnostic: request() is on
30+
// the base class and times out both the broker and the spawned transports.
31+
export const REQUEST_TIMEOUT_CODE = "EBROKERTIMEOUT";
3032
const BROKER_CONNECT_TIMEOUT_MS = 2000;
3133
const BROKER_INITIALIZE_TIMEOUT_MS = 5000;
34+
const SPAWNED_INITIALIZE_TIMEOUT_MS = 10000;
3235

3336
/** @type {ClientInfo} */
3437
const DEFAULT_CLIENT_INFO = {
@@ -89,18 +92,37 @@ class AppServerClientBase {
8992
* @template {AppServerMethod} M
9093
* @param {M} method
9194
* @param {import("./app-server-protocol").AppServerRequestParams<M>} params
95+
* @param {{ timeoutMs?: number, onTimeout?: () => void }} [options]
9296
* @returns {Promise<import("./app-server-protocol").AppServerResponse<M>>}
9397
*/
94-
request(method, params) {
98+
request(method, params, options = {}) {
9599
if (this.closed || this.exitResolved) {
96100
throw new Error("codex app-server client is closed.");
97101
}
98102

99103
const id = this.nextId;
100104
this.nextId += 1;
105+
const { timeoutMs, onTimeout } = options;
101106

102107
return new Promise((resolve, reject) => {
103-
this.pending.set(id, { resolve, reject, method });
108+
let timer = null;
109+
if (Number.isFinite(timeoutMs) && timeoutMs > 0) {
110+
timer = setTimeout(() => {
111+
// Reject and remove first; a late answer for this id is then a no-op
112+
// in handleLine (it checks this.pending.get(id)). unref'd so a timed-
113+
// out request can't pin the event loop. onTimeout (e.g. socket.destroy
114+
// for the broker transport) runs before reject to force-close the peer.
115+
if (this.pending.delete(id)) {
116+
try {
117+
onTimeout?.();
118+
} catch {}
119+
const error = /** @type {ProtocolError} */ (new Error(`codex app-server ${method} timed out.`));
120+
error.code = REQUEST_TIMEOUT_CODE;
121+
reject(error);
122+
}
123+
}, timeoutMs).unref?.();
124+
}
125+
this.pending.set(id, { resolve, reject, method, timer });
104126
this.sendMessage({ id, method, params });
105127
});
106128
}
@@ -147,6 +169,9 @@ class AppServerClientBase {
147169
return;
148170
}
149171
this.pending.delete(message.id);
172+
if (pending.timer) {
173+
clearTimeout(pending.timer);
174+
}
150175

151176
if (message.error) {
152177
pending.reject(createProtocolError(message.error.message ?? `codex app-server ${pending.method} failed.`, message.error));
@@ -177,6 +202,9 @@ class AppServerClientBase {
177202
this.exitError = error ?? null;
178203

179204
for (const pending of this.pending.values()) {
205+
if (pending.timer) {
206+
clearTimeout(pending.timer);
207+
}
180208
pending.reject(this.exitError ?? new Error("codex app-server connection closed."));
181209
}
182210
this.pending.clear();
@@ -230,13 +258,42 @@ class SpawnedCodexAppServerClient extends AppServerClientBase {
230258
this.handleLine(line);
231259
});
232260

233-
await this.request("initialize", {
234-
clientInfo: this.options.clientInfo ?? DEFAULT_CLIENT_INFO,
235-
capabilities: this.options.capabilities ?? DEFAULT_CAPABILITIES
236-
});
261+
try {
262+
await this.request("initialize", {
263+
clientInfo: this.options.clientInfo ?? DEFAULT_CLIENT_INFO,
264+
capabilities: this.options.capabilities ?? DEFAULT_CAPABILITIES
265+
}, {
266+
timeoutMs: this.options.spawnedInitializeTimeoutMs ?? SPAWNED_INITIALIZE_TIMEOUT_MS
267+
});
268+
} catch (error) {
269+
// A handshake failure (timeout, app-server crash) must not orphan the
270+
// spawned child. Kill it so the caller's fallback / teardown is clean.
271+
this.handleExit(error);
272+
this.killChildNow();
273+
throw error;
274+
}
237275
this.notify("initialized", {});
238276
}
239277

278+
// Best-effort synchronous kill of the spawned child. On Windows with
279+
// shell:true the direct child is cmd.exe, so terminateProcessTree takes the
280+
// whole tree (grandchild node included); elsewhere SIGTERM suffices. Shared
281+
// by initialize()'s handshake-failure path and close()'s deferred kill.
282+
killChildNow() {
283+
if (!(this.proc && !this.proc.killed && this.proc.exitCode === null)) {
284+
return;
285+
}
286+
try {
287+
if (process.platform === "win32") {
288+
terminateProcessTree(this.proc.pid);
289+
} else {
290+
this.proc.kill("SIGTERM");
291+
}
292+
} catch {
293+
// Best-effort cleanup — swallow to avoid crashing the host during shutdown.
294+
}
295+
}
296+
240297
async close() {
241298
if (this.closed) {
242299
await this.exitPromise;
@@ -251,23 +308,7 @@ class SpawnedCodexAppServerClient extends AppServerClientBase {
251308

252309
if (this.proc && !this.proc.killed) {
253310
this.proc.stdin.end();
254-
setTimeout(() => {
255-
if (this.proc && !this.proc.killed && this.proc.exitCode === null) {
256-
// On Windows with shell: true, the direct child is cmd.exe.
257-
// Use terminateProcessTree to kill the entire tree including
258-
// the grandchild node process.
259-
if (process.platform === "win32") {
260-
try {
261-
terminateProcessTree(this.proc.pid);
262-
} catch {
263-
// Best-effort cleanup inside an unref'd timer — swallow errors
264-
// to avoid crashing the host process during shutdown.
265-
}
266-
} else {
267-
this.proc.kill("SIGTERM");
268-
}
269-
}
270-
}, 50).unref?.();
311+
setTimeout(() => this.killChildNow(), 50).unref?.();
271312
}
272313

273314
await this.exitPromise;
@@ -292,8 +333,8 @@ class BrokerCodexAppServerClient extends AppServerClientBase {
292333

293334
async initialize() {
294335
const handshakeError = (message) => {
295-
const error = new Error(message);
296-
error.code = BROKER_HANDSHAKE_TIMEOUT_CODE;
336+
const error = /** @type {ProtocolError} */ (new Error(message));
337+
error.code = REQUEST_TIMEOUT_CODE;
297338
return error;
298339
};
299340

@@ -331,22 +372,18 @@ class BrokerCodexAppServerClient extends AppServerClientBase {
331372
});
332373

333374
// Bound the initialize request: a wedged broker may connect but never
334-
// answer. Race the request against a deadline; on timeout, force the
335-
// connection closed (handleExit rejects the pending request).
336-
let initializeTimer;
337-
const initializeDeadline = new Promise((_, reject) => {
338-
initializeTimer = setTimeout(() => {
339-
this.socket.destroy();
340-
reject(handshakeError("codex app-server initialize timed out."));
341-
}, initializeTimeoutMs).unref?.();
375+
// answer. On timeout, destroy the socket so the connection does not linger
376+
// (onTimeout) and reject with a handshake-timeout error code.
377+
await this.request("initialize", {
378+
clientInfo: this.options.clientInfo ?? DEFAULT_CLIENT_INFO,
379+
capabilities: this.options.capabilities ?? DEFAULT_CAPABILITIES
380+
}, {
381+
timeoutMs: initializeTimeoutMs,
382+
// Destroying the socket fires the close handler → handleExit, which
383+
// rejects any other pending requests and resolves exitPromise so a
384+
// wedged broker connection does not linger after the handshake fails.
385+
onTimeout: () => this.socket?.destroy()
342386
});
343-
await Promise.race([
344-
this.request("initialize", {
345-
clientInfo: this.options.clientInfo ?? DEFAULT_CLIENT_INFO,
346-
capabilities: this.options.capabilities ?? DEFAULT_CAPABILITIES
347-
}),
348-
initializeDeadline
349-
]).finally(() => clearTimeout(initializeTimer));
350387
this.notify("initialized", {});
351388
}
352389

@@ -400,8 +437,14 @@ export class CodexAppServerClient {
400437
// withAppServer cannot read `client.transport` (client is never assigned
401438
// there). Tag the error with the transport so the caller can still tell
402439
// a broker-handshake failure (worth a direct fallback) from a spawned one.
440+
// For the broker transport every initialization failure is transport-fatal
441+
// (timeout, ENOENT, ECONNREFUSED — the broker socket is unusable), so mark
442+
// it once here and let withAppServer match on a single predicate.
403443
if (error && !error.transport) {
404444
error.transport = client.transport;
445+
if (client.transport === "broker") {
446+
error.brokerFatal = true;
447+
}
405448
}
406449
throw error;
407450
}

‎plugins/codex/scripts/lib/codex.mjs‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ import os from "node:os";
4040
import path from "node:path";
4141

4242
import { readJsonFile } from "./fs.mjs";
43-
import { BROKER_BUSY_RPC_CODE, BROKER_ENDPOINT_ENV, BROKER_HANDSHAKE_TIMEOUT_CODE, CodexAppServerClient } from "./app-server.mjs";
43+
import { BROKER_BUSY_RPC_CODE, BROKER_ENDPOINT_ENV, CodexAppServerClient } from "./app-server.mjs";
4444
import { loadBrokerSession } from "./broker-lifecycle.mjs";
4545
import { binaryAvailable } from "./process.mjs";
4646
import { validateExplicitReasoningSelection, validateReasoningSelection } from "./model-catalog.mjs";
@@ -687,13 +687,16 @@ export async function withAppServer(cwd, fn, clientOptions = {}) {
687687
await client.close();
688688
return result;
689689
} catch (error) {
690-
const brokerRequested = client?.transport === "broker" || Boolean(process.env[BROKER_ENDPOINT_ENV]);
691-
const brokerHandshakeFailed =
692-
error?.code === BROKER_HANDSHAKE_TIMEOUT_CODE && (client?.transport === "broker" || error?.transport === "broker");
690+
// Fall back to a direct (non-broker) app-server when the broker transport
691+
// is unusable. Two distinct shapes:
692+
// - broker/busy (-32001) arrives AFTER a successful handshake, while
693+
// `client` is assigned — read the transport off the client.
694+
// - a handshake failure (timeout / ENOENT / ECONNREFUSED) rejects inside
695+
// connect() before `client` is assigned — tagged brokerFatal in
696+
// CodexAppServerClient.connect.
693697
const shouldRetryDirect =
694698
(client?.transport === "broker" && error?.rpcCode === BROKER_BUSY_RPC_CODE) ||
695-
brokerHandshakeFailed ||
696-
(brokerRequested && (error?.code === "ENOENT" || error?.code === "ECONNREFUSED"));
699+
error?.brokerFatal === true;
697700

698701
if (client) {
699702
await client.close().catch(() => {});

‎tests/broker-client-initialize-timeout.test.mjs‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import { CodexAppServerClient } from "../plugins/codex/scripts/lib/app-server.mj
55
import { withAppServer } from "../plugins/codex/scripts/lib/codex.mjs";
66
import { buildEnv, installFakeCodex } from "./fake-codex-fixture.mjs";
77
import { makeTempDir } from "./helpers.mjs";
8-
import { spawnWedgedServer } from "./broker-controller-helpers.mjs";
8+
import { spawnBusyBroker, spawnWedgedServer } from "./broker-controller-helpers.mjs";
99

1010
// Reproduces fork #29 / upstream openai#509: when the shared broker is wedged
1111
// (busy with a long-running request and never reaching the second socket), the
@@ -82,3 +82,46 @@ test("withAppServer falls back to a direct app-server when the broker wedges (#2
8282
wedged.cleanup();
8383
}
8484
});
85+
86+
// Drives the post-handshake busy fallback: the broker accepts initialize, then
87+
// returns BROKER_BUSY_RPC_CODE (-32001) for the operation's first request.
88+
// withAppServer must fall back to a direct app-server. Codex review of PR #32
89+
// caught that collapsing the OR-chain onto error.transport broke this path
90+
// (transport is tagged only on handshake failure, not on post-handshake busy).
91+
test("withAppServer falls back to a direct app-server on broker/busy after a successful handshake (#32)", async () => {
92+
const busy = await spawnBusyBroker();
93+
const cwd = makeTempDir();
94+
const binDir = makeTempDir();
95+
installFakeCodex(binDir);
96+
const env = buildEnv(binDir);
97+
98+
let calls = 0;
99+
try {
100+
const result = await Promise.race([
101+
withAppServer(
102+
cwd,
103+
async (client) => {
104+
calls += 1;
105+
// Any real RPC: on the broker client this hits broker/busy (-32001)
106+
// and must trigger the direct fallback; on the direct client it
107+
// resolves normally via the fake-codex app-server.
108+
await client.request("account/read", {});
109+
return { transport: client.transport };
110+
},
111+
{
112+
brokerEndpoint: `unix:${busy.socketPath}`,
113+
env
114+
}
115+
),
116+
new Promise((resolve) => setTimeout(() => resolve({ HANG: true }), 5000))
117+
]);
118+
119+
assert.ok(!result.HANG, "withAppServer hung instead of falling back to direct on broker/busy");
120+
// The broker attempt hits -32001, withAppServer retries on direct — so fn
121+
// runs at least once on the broker (which fails) and succeeds on direct.
122+
assert.ok(calls >= 1, "the operation must run via the direct fallback after broker/busy");
123+
assert.equal(result.transport, "direct", "the operation must complete on the direct transport after a busy broker");
124+
} finally {
125+
busy.cleanup();
126+
}
127+
});

‎tests/broker-controller-helpers.mjs‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,3 +231,62 @@ export async function spawnWedgedServer() {
231231
}
232232
};
233233
}
234+
235+
/**
236+
* A real net.createServer that speaks just enough JSON-RPC to pass the broker
237+
* handshake, then returns BROKER_BUSY_RPC_CODE (-32001) for every subsequent
238+
* request — simulates a shared broker that accepted initialize while busy with
239+
* another in-flight stream. Drives withAppServer's post-handshake busy
240+
* fallback, which must retry the operation on a direct (non-broker) client.
241+
*/
242+
export async function spawnBusyBroker() {
243+
const sessionDir = makeTempDir("broker-busy-");
244+
const socketPath = path.join(sessionDir, "broker.sock");
245+
const connections = new Set();
246+
const server = net.createServer((socket) => {
247+
socket.setEncoding("utf8");
248+
connections.add(socket);
249+
socket.on("close", () => connections.delete(socket));
250+
socket.on("error", () => {});
251+
let buffer = "";
252+
socket.on("data", (chunk) => {
253+
buffer += chunk;
254+
let newlineIndex = buffer.indexOf("\n");
255+
while (newlineIndex !== -1) {
256+
const line = buffer.slice(0, newlineIndex);
257+
buffer = buffer.slice(newlineIndex + 1);
258+
newlineIndex = buffer.indexOf("\n");
259+
let message;
260+
try {
261+
message = JSON.parse(line);
262+
} catch {
263+
continue;
264+
}
265+
if (message.id === undefined) {
266+
continue;
267+
}
268+
if (message.method === "initialize") {
269+
socket.write(`${JSON.stringify({ id: message.id, result: { userAgent: "busy-broker" } })}\n`);
270+
} else {
271+
socket.write(`${JSON.stringify({ id: message.id, error: { code: -32001, message: "Shared Codex broker is busy." } })}\n`);
272+
}
273+
}
274+
});
275+
});
276+
await new Promise((resolve) => server.listen(socketPath, resolve));
277+
return {
278+
socketPath,
279+
cleanup() {
280+
for (const socket of connections) {
281+
socket.destroy();
282+
}
283+
connections.clear();
284+
try {
285+
server.close();
286+
} catch {}
287+
try {
288+
fs.unlinkSync(socketPath);
289+
} catch {}
290+
}
291+
};
292+
}

‎tests/fake-codex-fixture.mjs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,12 @@ rl.on("line", (line) => {
321321
try {
322322
switch (message.method) {
323323
case "initialize":
324+
if (BEHAVIOR === "stalled-initialize") {
325+
// Never respond — simulates a spawned app-server that is alive (stdin
326+
// open) but wedged on initialize. SpawnedCodexAppServerClient.initialize()
327+
// must bound this handshake (fork #31) instead of hanging forever.
328+
break;
329+
}
324330
state.capabilities = message.params.capabilities || null;
325331
saveState(state);
326332
send({ id: message.id, result: { userAgent: "fake-codex-app-server" } });

0 commit comments

Comments
 (0)