Skip to content

Commit fbf45c8

Browse files
committed
feat(jsonl): manual port of upstream PR #24 + openai#311 (ANSI strip + non-JSONL guard)
본 fork 는 upstream `openai#24` (strip ANSI escape sequences) + `openai#311` (drop non-JSONL stdout garbage) 를 `git cherry-pick` 직접 불가 (fork v2.0.0/v2.1.0 hardening 의 broker/locale/TextDecoder 영역과 같은 file 같은 영역 변경으로 dry-run 시 `codex-companion.mjs` + `lib/codex.mjs` 양쪽 CONFLICT 확정). 두 PR 의 intent 를 v2.1.0 base 위에 manual port. 영역 — upstream `openai/codex-plugin-cc` issue #23 (JSONL parse: bracketed paste escape sequence in output) 해결. fork audit cycle (axis R5/R6 Tier 1 Group A) 의 highest priority cherry-pick 후보. ## 변경 - `plugins/codex/scripts/lib/jsonl.mjs` 신규 — `cleanProtocolLine(rawLine)` helper. PR openai#311 의 design (CSI + OSC + JSON prefix guard) 위에 PR #24 의 ECMA-48 완전 regex (CSI / OSC / DCS / SOS / PM / APC / simple Fp-Fe-Fs / lone ESC) 합성. 비-JSON line (예: zh-TW CP-950 mojibaked `taskkill SUCCESS` 메시지) silent drop — broker 연결 끊김 방지. - `plugins/codex/scripts/lib/app-server.mjs` `handleLine` — raw `JSON.parse` 앞에 `cleanProtocolLine` gate 추가. null 시 silent skip, candidate 시 parse 시도. - `plugins/codex/scripts/app-server-broker.mjs` JSONL pump — 동일 gate 적용. invalid JSON 은 기존 -32700 error response 유지. - `tests/jsonl.test.mjs` 신규 — PR openai#311 의 test 17건 통합 (CSI bracketed- paste / modifyOtherKeys / OSC title / shell integration / hyperlink / DCS-SOS-PM-APC string / Fp-Fe-Fs simple / mixed / Windows CP-950 mojibake / non-JSON prefix / nF intermediate / empty payload). ## 영향 - **upstream issue #23 (MED) 자동 해결**: parser raw `JSON.parse(line)` 가 zsh bracketed paste / OSC noise 로 broker 연결 끊기던 시나리오 차단. - **openai#171 wrong-citation 정정**: Round 4 가 "fork 이미 cover" 분류했으나 실제 ANSI strip 부재 확인 후 본 commit 으로 진짜 cover. - **openai#310 partial cover 완성**: 기존 locale mitigation 외에 parser 강건성 보강. ## 검증 - `node --test tests/jsonl.test.mjs` — 17/17 pass - fork 의 기존 streaming TextDecoder / CODEX_HOME isolation / locale override / MAX_BUFFERED_NOTIFICATIONS 동작 모두 보존 (handleLine 의 signature 만 변경, return 동작 동일) ## audit trail `docs/upstream-tracking/2026-05-18-upstream-backlog-audit.md` (axis R5 cross-link confirm + R6 wrong-citation 정정 + Tier 1 Group A).
1 parent 6f71507 commit fbf45c8

4 files changed

Lines changed: 224 additions & 6 deletions

File tree

‎plugins/codex/scripts/app-server-broker.mjs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import process from "node:process";
88
import { parseArgs } from "./lib/args.mjs";
99
import { BROKER_BUSY_RPC_CODE, CodexAppServerClient } from "./lib/app-server.mjs";
1010
import { parseBrokerEndpoint } from "./lib/broker-endpoint.mjs";
11+
import { cleanProtocolLine } from "./lib/jsonl.mjs";
1112

1213
const STREAMING_METHODS = new Set(["turn/start", "review/start", "thread/compact/start"]);
1314

@@ -245,13 +246,18 @@ async function main() {
245246
buffer = buffer.slice(newlineIndex + 1);
246247
newlineIndex = buffer.indexOf("\n");
247248

248-
if (!line.trim()) {
249+
// PR-G-A (#23 / upstream #24 + #311) — strip terminal/locale
250+
// noise (ANSI / OSC, Windows CP-950 taskkill prefix) before
251+
// JSON.parse so a single garbage byte from a TTY-inherited
252+
// subprocess does not poison the broker connection.
253+
const candidate = cleanProtocolLine(line);
254+
if (candidate === null) {
249255
continue;
250256
}
251257

252258
let message;
253259
try {
254-
message = JSON.parse(line);
260+
message = JSON.parse(candidate);
255261
} catch (error) {
256262
send(socket, {
257263
id: null,

‎plugins/codex/scripts/lib/app-server.mjs‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import { spawn } from "node:child_process";
1515
import readline from "node:readline";
1616
import { parseBrokerEndpoint } from "./broker-endpoint.mjs";
1717
import { ensureBrokerSession, loadBrokerSession } from "./broker-lifecycle.mjs";
18+
import { cleanProtocolLine } from "./jsonl.mjs";
1819
import { buildCommandInvocation, terminateProcessTree } from "./process.mjs";
1920

2021
const PLUGIN_MANIFEST_URL = new URL("../../.claude-plugin/plugin.json", import.meta.url);
@@ -353,16 +354,22 @@ class AppServerClientBase {
353354
}
354355
}
355356

356-
handleLine(line) {
357-
if (!line.trim()) {
357+
handleLine(rawLine) {
358+
// PR-G-A (#23 / upstream #24 + #311): drop terminal/locale noise
359+
// before JSON.parse. Empty lines, ANSI escape sequences (bracketed
360+
// paste, etc.) and localized OS prefixes (CP-950 mojibaked Windows
361+
// taskkill output) used to tear the connection down via handleExit.
362+
// See lib/jsonl.mjs for the full rationale.
363+
const candidate = cleanProtocolLine(rawLine);
364+
if (candidate === null) {
358365
return;
359366
}
360367

361368
let message;
362369
try {
363-
message = JSON.parse(line);
370+
message = JSON.parse(candidate);
364371
} catch (error) {
365-
this.handleExit(createProtocolError(`Failed to parse codex app-server JSONL: ${error.message}`, { line }));
372+
this.handleExit(createProtocolError(`Failed to parse codex app-server JSONL: ${error.message}`, { line: rawLine }));
366373
return;
367374
}
368375

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
/**
2+
* Helpers for reading the Codex app-server JSONL protocol.
3+
*
4+
* Manual port of upstream PR #311 (drop non-JSONL garbage on stdout)
5+
* fused with the more complete ANSI grammar from upstream PR #24
6+
* (full ECMA-48 escape repertoire). Both upstream PRs target the same
7+
* `JSON.parse(line)` call sites in `app-server.mjs` + `app-server-broker.mjs`
8+
* but neither could be cherry-picked cleanly onto the v2.1.0 base
9+
* because the surrounding broker / locale / TextDecoder code diverged
10+
* substantially. See `docs/upstream-tracking/2026-05-18-upstream-backlog-audit.md`
11+
* (axis R6 / Tier 1 Group A) for the audit trail.
12+
*
13+
* The codex CLI's app-server occasionally emits non-JSON bytes on stdout
14+
* that have nothing to do with the protocol. Two known sources:
15+
*
16+
* 1. Terminal / shell init noise — e.g. zsh writes the bracketed-paste
17+
* marker `\x1b[?2004h` when a subprocess inherits the parent's TTY
18+
* (issue #23).
19+
*
20+
* 2. Localized OS messages on Windows non-English locales. On zh-TW
21+
* (CP-950 / Big5), when `codex.exe` runs `taskkill /T /F` to clean
22+
* up a failed MCP child and inadvertently routes taskkill's stdout
23+
* to its own stdout, we see the bytes `A6 A8 A5 5C 3A 20 50 49 44 ...`
24+
* ("成功: PID 為 xxxx ...") which decode under UTF-8 as
25+
* `���\: PID ...`.
26+
*
27+
* Both cases historically tore the broker connection down with
28+
* `Failed to parse codex app-server JSONL: Unexpected token …`. The
29+
* client never recovered because `handleExit` rejects every in-flight
30+
* request before any subsequent valid record arrives.
31+
*
32+
* `cleanProtocolLine` is the conservative guard:
33+
*
34+
* - Strip ANSI escape sequences (CSI / OSC / DCS / SOS / PM / APC /
35+
* simple Fp-Fe-Fs escapes / lone ESC).
36+
* - Trim whitespace.
37+
* - Require the first remaining character to be `{` or `[`. JSONL
38+
* records cannot start with anything else, so any other prefix is
39+
* definitively garbage and is dropped.
40+
*
41+
* Lines that pass these checks are still parsed with `JSON.parse`; if
42+
* that fails the caller surfaces a real protocol error as before.
43+
*/
44+
45+
// ANSI escape grammar from upstream PR #24 (ECMA-48 complete):
46+
//
47+
// CSI: ESC [ <param bytes 0x30-0x3F>* <intermediate bytes 0x20-0x2F>* <final 0x40-0x7E>
48+
// param bytes include digits, ;, :, <, =, >, ? — not just [0-9;?]
49+
// e.g. ESC[?2004h (bracketed paste), ESC[>4;2m (modifyOtherKeys), ESC[200~ (paste wrapper)
50+
// OSC: ESC ] <any> (BEL | ESC \)
51+
// e.g. ESC]0;title BEL (terminal title), ESC]133;A BEL (shell integration)
52+
// String: ESC [P|X|^|_] <any> (BEL | ESC \) — DCS / SOS / PM / APC
53+
// Simple: ESC <intermediate bytes 0x20-0x2F>* <final byte 0x30-0x7E>
54+
// Fp (0x30-0x3F): ESC 7/8 (save/restore cursor), ESC = / ESC > (keypad)
55+
// Fe (0x40-0x5F): ESC c (reset), ESC M (reverse index), ESC D/E
56+
// Fs (0x60-0x7E): standardized single functions
57+
// Lone: bare ESC fallback (e.g. truncated sequences)
58+
//
59+
// eslint-disable-next-line no-control-regex
60+
const ANSI_ESCAPE_RE =
61+
/\x1b\[[\x30-\x3f]*[\x20-\x2f]*[\x40-\x7e]|\x1b[PX^_].*?(?:\x07|\x1b\\)|\x1b\].*?(?:\x07|\x1b\\)|\x1b[\x20-\x2f]*[\x30-\x7e]|\x1b/g;
62+
63+
/**
64+
* Returns a JSON-shaped candidate string for the given raw line, or
65+
* `null` if the line should be skipped because it cannot be valid JSONL.
66+
*
67+
* Accepts arbitrary input — non-string values short-circuit to `null` so
68+
* callers don't have to type-check before invoking.
69+
*
70+
* @param {unknown} rawLine
71+
* @returns {string | null}
72+
*/
73+
export function cleanProtocolLine(rawLine) {
74+
if (typeof rawLine !== "string") {
75+
return null;
76+
}
77+
const cleaned = rawLine.replace(ANSI_ESCAPE_RE, "").trim();
78+
if (!cleaned) {
79+
return null;
80+
}
81+
const firstChar = cleaned.charCodeAt(0);
82+
if (firstChar !== 0x7b /* { */ && firstChar !== 0x5b /* [ */) {
83+
return null;
84+
}
85+
return cleaned;
86+
}

‎tests/jsonl.test.mjs‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
import test from "node:test";
2+
import assert from "node:assert/strict";
3+
4+
import { cleanProtocolLine } from "../plugins/codex/scripts/lib/jsonl.mjs";
5+
6+
// Manual port of upstream PR #24 + PR #311 tests, merged onto the v2.1.0
7+
// base. The fork could not cherry-pick either PR cleanly (broker / locale
8+
// hardening diverged), so the helper lives in `lib/jsonl.mjs` and the test
9+
// suite covers both ANSI-strip (PR #24) and JSONL-prefix guard (PR #311).
10+
// Audit trail: docs/upstream-tracking/2026-05-18-upstream-backlog-audit.md
11+
// (axis R6 / Tier 1 Group A).
12+
13+
test("cleanProtocolLine returns null for empty / whitespace lines", () => {
14+
assert.equal(cleanProtocolLine(""), null);
15+
assert.equal(cleanProtocolLine(" "), null);
16+
assert.equal(cleanProtocolLine("\t\r\n"), null);
17+
});
18+
19+
test("cleanProtocolLine returns null for non-string input", () => {
20+
assert.equal(cleanProtocolLine(undefined), null);
21+
assert.equal(cleanProtocolLine(null), null);
22+
assert.equal(cleanProtocolLine(42), null);
23+
});
24+
25+
test("cleanProtocolLine passes through plain JSON object lines unchanged", () => {
26+
assert.equal(cleanProtocolLine('{"id":1,"result":{}}'), '{"id":1,"result":{}}');
27+
assert.equal(cleanProtocolLine(' {"a":1} '), '{"a":1}');
28+
});
29+
30+
test("cleanProtocolLine passes through plain JSON array lines unchanged", () => {
31+
assert.equal(cleanProtocolLine("[1,2,3]"), "[1,2,3]");
32+
});
33+
34+
test("cleanProtocolLine strips bracketed-paste-mode ANSI prefix (issue #23)", () => {
35+
assert.equal(cleanProtocolLine('\x1b[?2004h{"id":1}'), '{"id":1}');
36+
assert.equal(cleanProtocolLine('{"id":1}\x1b[?2004l'), '{"id":1}');
37+
});
38+
39+
test("cleanProtocolLine strips CSI sequences whose final byte is not a letter", () => {
40+
// Bracketed-paste content markers use `~` (0x7E) as the final byte.
41+
// ECMA-48 CSI allows any final byte in 0x40..0x7E.
42+
assert.equal(cleanProtocolLine('\x1b[200~{"id":1}\x1b[201~'), '{"id":1}');
43+
assert.equal(cleanProtocolLine('\x1b[@{"id":1}'), '{"id":1}');
44+
assert.equal(cleanProtocolLine('\x1b[`{"id":1}'), '{"id":1}');
45+
});
46+
47+
test("cleanProtocolLine strips modifyOtherKeys (`>` parameter byte)", () => {
48+
assert.equal(cleanProtocolLine('\x1b[>4;2m{"id":1}'), '{"id":1}');
49+
assert.equal(cleanProtocolLine('\x1b[>4;0m{"id":1}'), '{"id":1}');
50+
});
51+
52+
test("cleanProtocolLine strips OSC window-title sequences (BEL + ST terminator)", () => {
53+
assert.equal(cleanProtocolLine('\x1b]0;My Terminal\x07{"id":1}'), '{"id":1}');
54+
assert.equal(cleanProtocolLine('\x1b]0;My Terminal\x1b\\{"id":1}'), '{"id":1}');
55+
});
56+
57+
test("cleanProtocolLine strips iTerm2/kitty shell-integration sequences", () => {
58+
assert.equal(cleanProtocolLine('\x1b]133;A\x07{"id":1}'), '{"id":1}');
59+
assert.equal(cleanProtocolLine('\x1b]133;D;0\x07{"id":1}'), '{"id":1}');
60+
});
61+
62+
test("cleanProtocolLine strips OSC hyperlink sequences", () => {
63+
assert.equal(cleanProtocolLine('\x1b]8;params;uri\x07{"id":1}'), '{"id":1}');
64+
});
65+
66+
test("cleanProtocolLine strips DCS / SOS / PM / APC string sequences", () => {
67+
assert.equal(cleanProtocolLine('\x1bPfoo=bar\x1b\\{"id":1}'), '{"id":1}');
68+
assert.equal(cleanProtocolLine('\x1b_Gfoo\x1b\\{"id":1}'), '{"id":1}');
69+
assert.equal(cleanProtocolLine('\x1b^hello\x1b\\{"id":1}'), '{"id":1}');
70+
assert.equal(cleanProtocolLine('\x1bXdata\x1b\\{"id":1}'), '{"id":1}');
71+
});
72+
73+
test("cleanProtocolLine strips simple Fp / Fe / Fs escapes", () => {
74+
// Fp finals 0x30-0x3F (e.g. ESC 7 save cursor, ESC = keypad mode)
75+
assert.equal(cleanProtocolLine('\x1b7{"id":1}'), '{"id":1}');
76+
assert.equal(cleanProtocolLine('\x1b={"id":1}'), '{"id":1}');
77+
// Fe finals 0x40-0x5F (e.g. ESC M reverse index, ESC E next line)
78+
assert.equal(cleanProtocolLine('\x1bM{"id":1}'), '{"id":1}');
79+
assert.equal(cleanProtocolLine('\x1bE{"id":1}'), '{"id":1}');
80+
// Fs finals 0x60-0x7E (e.g. ESC c RIS full reset)
81+
assert.equal(cleanProtocolLine('\x1bc{"id":1}'), '{"id":1}');
82+
});
83+
84+
test("cleanProtocolLine handles mixed ANSI noise around a JSON record", () => {
85+
const line = '\x1b]0;zsh\x07\x1b[1;34m{"jsonrpc":"2.0","id":1}\x1b[0m';
86+
assert.equal(cleanProtocolLine(line), '{"jsonrpc":"2.0","id":1}');
87+
});
88+
89+
test("cleanProtocolLine drops localized Windows taskkill prefix (issue #23 sibling)", () => {
90+
// CP-950 / Big5 mojibake of `taskkill /T /F` success message on zh-TW.
91+
// Not a valid JSON record — should be dropped silently rather than
92+
// tearing the broker connection down.
93+
assert.equal(cleanProtocolLine("���\\: PID 1234"), null);
94+
});
95+
96+
test("cleanProtocolLine drops non-JSON prefix even when ANSI stripped to plain text", () => {
97+
// After ANSI strip the line still does not start with `{` or `[`.
98+
assert.equal(cleanProtocolLine("\x1b[1;34mhello world"), null);
99+
assert.equal(cleanProtocolLine("trailing log line without JSON"), null);
100+
});
101+
102+
// NOTE: `\x1b{...}` (bare ESC immediately followed by JSON start `{`) is a
103+
// theoretical corner case — in practice no terminal emitter produces ESC
104+
// directly before `{`. The simple-escape branch of the regex
105+
// (`\x1b[\x20-\x2f]*[\x30-\x7e]`) matches ECMA-48 escapes whose final byte
106+
// is in 0x30-0x7E, which includes `{` (0x7B), so this input would lose the
107+
// opening brace. JSON prefix guard then drops the line as garbage rather
108+
// than producing a malformed JSON. Acceptable trade-off vs the cost of
109+
// rewriting the regex to carve out `{` / `[` from the final byte class.
110+
111+
test("cleanProtocolLine: nF escape with intermediate byte stripped", () => {
112+
// ESC space F: intermediate 0x20, final 0x46.
113+
assert.equal(cleanProtocolLine('\x1b F{"id":1}'), '{"id":1}');
114+
});
115+
116+
test("cleanProtocolLine: empty payload after strip returns null", () => {
117+
assert.equal(cleanProtocolLine("\x1b[0m"), null);
118+
assert.equal(cleanProtocolLine("\x1b[?2004h\x1b[?2004l"), null);
119+
});

0 commit comments

Comments
 (0)