Skip to content

Commit 544f1c5

Browse files
geunwoojun99claudelidge-jun
authored
fix(jev): exclude Claude Code reminders from decision state (#6662)
Treat Claude Code system-reminder blocks as protected envelopes when building JEV decision state. Long injected guidance no longer displaces the real request in the task sample. Adds regression tests and updates the guide and structure contract. Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: JUN <[email protected]>
1 parent 36fa092 commit 544f1c5

4 files changed

Lines changed: 141 additions & 5 deletions

File tree

‎docs-site/src/content/docs/guides/combos.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -474,9 +474,11 @@ The decision state is deliberately bounded: up to 500 characters of the current
474474
240-character previous-assistant tail, a 520-character latest-tool-output tail, the tool name, and
475475
boolean image/tool signals may be sent to the selected decision backend. It excludes credentials,
476476
request headers, raw image bytes, tool arguments, encrypted reasoning, and full conversation history.
477-
Use a decision method only for content you are willing to send to that backend. Recognized OpenCodex machine-context
478-
envelopes are removed from all three text samples, but ordinary assistant and tool-output text is
479-
not a secret scanner and may still contain sensitive content. TypeSafe states that Jev is not
477+
Use a decision method only for content you are willing to send to that backend. Recognized Codex machine-context
478+
envelopes and Claude Code `<system-reminder>` blocks are removed from all three text samples before
479+
clipping, so injected reminders do not displace the actual task. The original model request is
480+
unchanged. This is not a secret scanner: ordinary assistant and tool-output text may still contain
481+
sensitive content. TypeSafe states that Jev is not
480482
trained on customer requests, but its terms set no fixed retention period for submitted state and
481483
offer zero data retention only on enterprise plans
482484
([models](https://docs.typesafe.ai/models), [legal](https://docs.typesafe.ai/legal)). TypeSafe

‎src/combos/jev.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ const TOOL_NAME_CHARS = 160;
4747
const VISIBLE_TEXT_CHUNK_CHARS = 16_384;
4848

4949
const ENVELOPE_TAGS = [
50+
"system-reminder",
5051
"codex_internal_context",
5152
"recommended_plugins",
5253
"environment_context",
@@ -302,7 +303,8 @@ function taskWithoutProtectedEnvelopes(text: string): string {
302303
}
303304
} else {
304305
if (stack.length === 0) appendTrimmedRange(visible, "\n", 0, 1);
305-
if (!completedGoal && !goal && tag === "codex_internal_context") {
306+
// A Codex goal quoted inside a Claude reminder is context, never a fallback task.
307+
if (!completedGoal && !goal && tag === "codex_internal_context" && !stack.includes("system-reminder")) {
306308
goal = { sample: emptyTextSample(), pendingWhitespace: emptyTextSample() };
307309
goalDepth = stack.length;
308310
}

‎structure/providers/jev-decision.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,11 @@ the usage aggregate reports per-backend counts and latency with older rows as `u
5454

5555
`src/combos/jev.ts` extracts bounded user-task, previous-assistant, and latest-tool-output text plus
5656
the tool name and boolean signals; raw image data, tool arguments, encrypted reasoning, headers, and
57-
the JEV credential are excluded. It owns the joint target/effort choice map, strict response
57+
the JEV credential are excluded. All three text samples omit recognized Codex protected envelopes
58+
and Claude Code `<system-reminder>` blocks before clipping, including nested and unclosed blocks.
59+
Only an envelope-only `codex_internal_context` goal outside a reminder may supply a fallback task;
60+
reminder-only text supplies no task. Reminder-free inputs keep their existing sampling behavior.
61+
It owns the joint target/effort choice map, strict response
5862
validation, canonical `jev-latest` destination, default four-second deadline, no-redirect policy, bounded response,
5963
and caller-cancellation propagation. Missing credentials or safe state, transport failures, and invalid
6064
answers fail open to the first eligible target; no response can escape the configured choice map.

‎tests/routing/jev-decision.test.ts‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import {
1010
type JevCandidate,
1111
type ResolveJevDecisionOptions,
1212
} from "../../src/combos/jev";
13+
import { anthropicToResponsesBody } from "../../src/claude/inbound";
1314
import type { OcxConfig } from "../../src/types";
1415

1516
const candidates: JevCandidate[] = [
@@ -138,6 +139,133 @@ describe("JEV bounded decision state", () => {
138139
expect(JSON.stringify(state)).not.toContain("TOOL_MACHINE_SECRET");
139140
});
140141

142+
test.each(["string", "blocks"])("extracts the real first Claude Code task from %s content", shape => {
143+
const task = "Fix the parser and add a regression test.";
144+
const reminders = [
145+
`<system-reminder>Project CLAUDE.md guidance\n${"Follow project conventions.\n".repeat(200)}</system-reminder>`,
146+
"<system-reminder>User context and attribution guidance.</system-reminder>",
147+
];
148+
const content = shape === "string"
149+
? [...reminders, task].join("\n\n")
150+
: [...reminders, task].map(text => ({ type: "text", text }));
151+
const body = anthropicToResponsesBody({
152+
model: "claude-sonnet-4-6",
153+
messages: [{ role: "user", content }],
154+
});
155+
const original = JSON.stringify(body);
156+
157+
expect(original).toContain("<system-reminder>");
158+
expect(buildJevState(body)).toEqual({
159+
task,
160+
signals: { has_image: false, tool_history: false },
161+
step: { type: "user_turn" },
162+
});
163+
expect(JSON.stringify(body)).toBe(original);
164+
});
165+
166+
test("strips reminders from the latest task and preserves head/tail clipping", () => {
167+
const task = `${"h".repeat(380)}${"t".repeat(380)}`;
168+
const body = anthropicToResponsesBody({
169+
model: "claude-sonnet-4-6",
170+
messages: [
171+
{ role: "user", content: "An older request." },
172+
{ role: "assistant", content: "Ready for the next request." },
173+
{
174+
role: "user",
175+
content: `<system-reminder>${"context".repeat(40_000)}</system-reminder>\n${task}`,
176+
},
177+
],
178+
});
179+
180+
expect(buildJevState(body)).toEqual({
181+
task: `${"h".repeat(320)}\n[...]\n${"t".repeat(173)}`,
182+
signals: { has_image: false, tool_history: false },
183+
step: { type: "user_turn" },
184+
previous_assistant: "Ready for the next request.",
185+
});
186+
});
187+
188+
test("uses protected nesting semantics without salvaging reminder contents as a Codex goal", () => {
189+
const reminder = '<system-reminder><codex_internal_context source="goal">Quoted goal</codex_internal_context></system-reminder>';
190+
expect(buildJevState({ input: reminder })).toMatchObject({ task: "" });
191+
expect(buildJevState({
192+
input: `<codex_internal_context source="goal">Active goal${reminder}</codex_internal_context>`,
193+
})).toMatchObject({ task: "Active goal" });
194+
expect(buildJevState({
195+
input: "<system-reminder>Outer<system-reminder>Inner</system-reminder>Outer</system-reminder>Real task",
196+
})).toMatchObject({ task: "Real task" });
197+
expect(buildJevState({
198+
input: "Visible task<system-reminder>Unclosed context",
199+
})).toMatchObject({ task: "Visible task" });
200+
expect(buildJevState({ input: "<system-reminder>Unclosed context" })).toMatchObject({ task: "" });
201+
expect(buildJevState({ input: "Before<system-reminder>Context</system-reminder>after" }))
202+
.toMatchObject({ task: "Before\nafter" });
203+
});
204+
205+
test("removes Claude Code reminders from converted assistant and tool-result tails", () => {
206+
const reminder = `<system-reminder>${"Context only. ".repeat(200)}</system-reminder>`;
207+
const body = anthropicToResponsesBody({
208+
model: "claude-sonnet-4-6",
209+
messages: [
210+
{ role: "user", content: "Continue the investigation." },
211+
{
212+
role: "assistant",
213+
content: [
214+
{ type: "text", text: `Visible plan${reminder}visible follow-up${reminder}` },
215+
{ type: "tool_use", id: "call-1", name: "Read", input: { file: "example.ts" } },
216+
],
217+
},
218+
{
219+
role: "user",
220+
content: [{ type: "tool_result", tool_use_id: "call-1", content: `Visible result${reminder}` }],
221+
},
222+
],
223+
});
224+
225+
expect(buildJevState(body)).toEqual({
226+
task: "Continue the investigation.",
227+
signals: { has_image: false, tool_history: true },
228+
step: { type: "tool_step", last_tool_output_tail: "Visible result", tool_call: { name: "Read" } },
229+
previous_assistant: "Visible plan\nvisible follow-up",
230+
});
231+
});
232+
233+
test("keeps reminder-free states byte-identical to the pre-reminder behavior", () => {
234+
const task = ` ${"h".repeat(380)}<example>ordinary markup</example>${"t".repeat(380)} `;
235+
const body = anthropicToResponsesBody({
236+
model: "claude-sonnet-4-6",
237+
messages: [
238+
{ role: "user", content: task },
239+
{
240+
role: "assistant",
241+
content: [
242+
{ type: "text", text: ` ${"a".repeat(300)}<example>plan</example> ` },
243+
{ type: "tool_use", id: "call-1", name: "Read", input: { file: "example.ts" } },
244+
],
245+
},
246+
{
247+
role: "user",
248+
content: [{ type: "tool_result", tool_use_id: "call-1", content: ` ${"z".repeat(600)} ` }],
249+
},
250+
],
251+
});
252+
const expected = {
253+
task: `${"h".repeat(320)}\n[...]\n${"t".repeat(173)}`,
254+
signals: { has_image: false, tool_history: true },
255+
step: { type: "tool_step", last_tool_output_tail: "z".repeat(520), tool_call: { name: "Read" } },
256+
previous_assistant: `${"a".repeat(217)}<example>plan</example>`,
257+
};
258+
259+
expect(JSON.stringify(buildJevState(body))).toBe(JSON.stringify(expected));
260+
expect(JSON.stringify(buildJevState({ input: task }))).toBe(JSON.stringify({
261+
task: expected.task,
262+
signals: { has_image: false, tool_history: false },
263+
step: { type: "user_turn" },
264+
}));
265+
expect(buildJevState({ input: "<environment_context>Context</environment_context>Codex task" }))
266+
.toMatchObject({ task: "Codex task" });
267+
});
268+
141269
test("salvages an envelope-only active goal but drops catalog-only envelopes", () => {
142270
expect(buildJevState({
143271
input: '<codex_internal_context source="goal">Keep implementing JEV.</codex_internal_context>',

0 commit comments

Comments
 (0)