Skip to content

Commit a955a2c

Browse files
test(server): record Codex multi-agent V2 subagents live on 0.156.1
The subagent_v2 and subagent_v2_nested transcripts were hand-built with synthetic thread ids and hand-ordered frames. Record both against a real Codex 0.156.1 app-server on gpt-5.6-sol, which runs multi-agent v2. The recorder gains the two scenarios and a per-run thread/start config override: the nested scenario raises agents.max_depth to 3, since Codex only lets root spawn by default. Replay ignores thread/start config, so the transcript still matches the adapter's frames. The prompts now ask for the spawns explicitly; "just say hello" alone no longer spawns anything on this model. In both recordings Codex emits the child's idle status, then the parent's subAgentActivity(started), then the child's turn/started, so the adapter registers the child before its first turn. Assertions now name the agent paths and answers the model chose, and the scenario expectations use the recorder's method labels. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent e37e762 commit a955a2c

9 files changed

Lines changed: 220 additions & 89 deletions

File tree

‎apps/server/scripts/record-codex-app-server-replay-fixture.ts‎

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ import {
2828
SUBAGENT_CONTINUE_PARENT_PROMPT,
2929
SUBAGENT_CONTINUE_PROMPT,
3030
SUBAGENT_PROMPT,
31+
SUBAGENT_V2_LIVE_PROMPT,
32+
SUBAGENT_V2_NESTED_LIVE_PROMPT,
3133
THREAD_ROLLBACK_AFTER_PROMPT,
3234
THREAD_ROLLBACK_FIRST_PROMPT,
3335
THREAD_ROLLBACK_SECOND_PROMPT,
@@ -79,6 +81,8 @@ const SCENARIO_NAMES = [
7981
"tool_call_restricted_granular",
8082
"subagent",
8183
"subagent_continue",
84+
"subagent_v2",
85+
"subagent_v2_nested",
8286
"multi_turn",
8387
"provider_thread_resume",
8488
"todo_list",
@@ -109,6 +113,8 @@ interface ReplayRun {
109113
readonly description: string;
110114
readonly steps: ReadonlyArray<ReplayStep>;
111115
readonly turnDefaults?: Omit<TurnStartParams, "input" | "threadId">;
116+
/** Config overrides for `thread/start`; replay ignores them when matching frames. */
117+
readonly threadConfig?: CodexSchema.V2ThreadStartParams["config"];
112118
}
113119

114120
type ReplayStep =
@@ -426,6 +432,41 @@ function scenarios(): ReadonlyArray<ReplayScenario> {
426432
},
427433
],
428434
},
435+
{
436+
name: "subagent_v2",
437+
fileName: "subagent_v2.ndjson",
438+
description:
439+
"One root turn on a multi-agent v2 model that spawns one native subagent and waits for it.",
440+
runs: [
441+
{
442+
name: "spawn-v2-subagent",
443+
description:
444+
"Record with a v2 model (e.g. --model gpt-5.6-sol) so Codex emits subAgentActivity items.",
445+
steps: [{ type: "turn", label: "spawn-v2-subagent", prompt: SUBAGENT_V2_LIVE_PROMPT }],
446+
},
447+
],
448+
},
449+
{
450+
name: "subagent_v2_nested",
451+
fileName: "subagent_v2_nested.ndjson",
452+
description:
453+
"One root turn on a multi-agent v2 model whose subagent spawns a subagent that spawns a leaf.",
454+
runs: [
455+
{
456+
name: "spawn-nested-v2-subagents",
457+
description:
458+
"Record with a v2 model (e.g. --model gpt-5.6-sol); depth 3 lets each child spawn again.",
459+
threadConfig: { "agents.max_depth": 3 },
460+
steps: [
461+
{
462+
type: "turn",
463+
label: "spawn-nested-v2-subagents",
464+
prompt: SUBAGENT_V2_NESTED_LIVE_PROMPT,
465+
},
466+
],
467+
},
468+
],
469+
},
429470
{
430471
name: "subagent_continue",
431472
fileName: "subagent_continue.ndjson",
@@ -1218,7 +1259,10 @@ function runReplaySession({
12181259

12191260
yield* Effect.gen(function* () {
12201261
const client = yield* initializeClient;
1221-
const thread = yield* client.request("thread/start", {});
1262+
const thread = yield* client.request(
1263+
"thread/start",
1264+
run.threadConfig === undefined ? {} : { config: run.threadConfig },
1265+
);
12221266
let activeThreadId = thread.thread.id;
12231267
const threadIds = new Map<string, string>([["source", thread.thread.id]]);
12241268

‎apps/server/src/orchestration-v2/testkit/CodexReplayFixtures.integration.test.ts‎

Lines changed: 2 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -168,30 +168,14 @@ const scenarioExpectations = {
168168
},
169169
subagent_v2: {
170170
outgoing: ["initialize", "initialized", "thread/start", "turn/start"],
171-
incoming: [
172-
"turn/started",
173-
"item/completed/subAgentActivity-started",
174-
"turn/started/child",
175-
"item/completed/child-answer",
176-
"turn/completed",
177-
],
171+
incoming: ["turn/started", "item/completed", "item/agentMessage/delta", "turn/completed"],
178172
turnStartCount: 1,
179173
turnCompletedCount: 2,
180174
approvalRequestCount: 0,
181175
},
182176
subagent_v2_nested: {
183177
outgoing: ["initialize", "initialized", "thread/start", "turn/start"],
184-
incoming: [
185-
"turn/started",
186-
"item/completed/root-subagent",
187-
"turn/started/child",
188-
"item/completed/child-subagent",
189-
"turn/started/grandchild",
190-
"item/completed/grandchild-subagent",
191-
"turn/started/leaf",
192-
"item/completed/leaf-answer",
193-
"turn/completed",
194-
],
178+
incoming: ["turn/started", "item/completed", "item/agentMessage/delta", "turn/completed"],
195179
turnStartCount: 1,
196180
turnCompletedCount: 4,
197181
approvalRequestCount: 0,

‎apps/server/src/orchestration-v2/testkit/fixtures/index.ts‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ import { assertCursorSubagentOutput } from "./subagent/cursor_output.ts";
4444
import { assertSubagentContinueOutput } from "./subagent_continue/codex_output.ts";
4545
import { subagentContinueInput } from "./subagent_continue/input.ts";
4646
import { assertSubagentV2Output } from "./subagent_v2/codex_output.ts";
47-
import { subagentV2Input } from "./subagent_v2/input.ts";
47+
import { subagentV2Input, subagentV2NestedInput } from "./subagent_v2/input.ts";
4848
import { assertSubagentV2NestedOutput } from "./subagent_v2_nested/codex_output.ts";
4949
import { assertClaudeThreadRollbackOutput } from "./thread_rollback/claude_output.ts";
5050
import { assertThreadRollbackOutput } from "./thread_rollback/codex_output.ts";
@@ -439,19 +439,21 @@ export const ORCHESTRATOR_REPLAY_FIXTURES: ReadonlyArray<OrchestratorReplayFixtu
439439
{
440440
driver: ProviderDriverKind.make("codex"),
441441
transcriptFile: new URL("./subagent_v2/codex_transcript.ndjson", import.meta.url),
442-
modelSelection: CODEX_MODEL_SELECTION,
442+
// Recorded live on Codex 0.156.1; gpt-5.6-sol runs multi-agent v2.
443+
modelSelection: { ...CODEX_MODEL_SELECTION, model: "gpt-5.6-sol" },
443444
assertOutput: assertSubagentV2Output,
444445
},
445446
],
446447
},
447448
{
448449
name: "subagent_v2_nested",
449-
buildInput: subagentV2Input,
450+
buildInput: subagentV2NestedInput,
450451
providers: [
451452
{
452453
driver: ProviderDriverKind.make("codex"),
453454
transcriptFile: new URL("./subagent_v2_nested/codex_transcript.ndjson", import.meta.url),
454-
modelSelection: CODEX_MODEL_SELECTION,
455+
// Recorded live on Codex 0.156.1; gpt-5.6-sol runs multi-agent v2.
456+
modelSelection: { ...CODEX_MODEL_SELECTION, model: "gpt-5.6-sol" },
455457
assertOutput: assertSubagentV2NestedOutput,
456458
},
457459
],

‎apps/server/src/orchestration-v2/testkit/fixtures/shared.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,10 @@ export const MESSAGE_STEERING_INITIAL_PROMPT =
4747
"Respond with exactly: steering fixture initial response";
4848
export const SUBAGENT_PROMPT =
4949
"Spawn 2 subagents, one to read package.json and one to read tsconfig.json";
50-
export const SUBAGENT_V2_PROMPT = "just say hello";
50+
export const SUBAGENT_V2_NESTED_LIVE_PROMPT =
51+
"Spawn one subagent and tell it to spawn its own subagent, which must in turn spawn one more subagent whose only task is to reply with exactly: Hello. Each agent waits for its child and replies with exactly what the child said. Wait for your subagent, then reply with exactly what it said.";
52+
export const SUBAGENT_V2_LIVE_PROMPT =
53+
"Spawn one subagent whose only task is to reply with exactly: Hello. Wait for it to finish, then reply with exactly what it said.";
5154
export const OPENCODE_SUBAGENT_PROMPT =
5255
"Use the task tool exactly once. Delegate to the general subagent with this prompt: Respond exactly CHILD_OK. After the task completes, respond exactly PARENT_OK.";
5356
export const SUBAGENT_CONTINUE_PROMPT =

‎apps/server/src/orchestration-v2/testkit/fixtures/subagent_v2/codex_output.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ import {
1111
assertTurnItemTypes,
1212
assertUserMessagesInclude,
1313
projectionFor,
14-
SUBAGENT_V2_PROMPT,
14+
SUBAGENT_V2_LIVE_PROMPT,
1515
} from "../shared.ts";
1616

1717
export function assertSubagentV2Output(
@@ -31,15 +31,15 @@ export function assertSubagentV2Output(
3131
assertExecutionNodeKinds(projection, ["root_turn", "subagent"]);
3232
assertRunProviderTurnCardinality({ projection, rootRunCount: 1 });
3333
assertNoExtraAppRunsForProviderChildren({ projection, expectedAppRuns: 1 });
34-
assertUserMessagesInclude(projection, [SUBAGENT_V2_PROMPT]);
34+
assertUserMessagesInclude(projection, [SUBAGENT_V2_LIVE_PROMPT]);
3535
assert.lengthOf(projection.subagents, 1);
3636
assert.lengthOf(result.shellSnapshot.threads, 2);
3737

3838
const subagent = projection.subagents[0]!;
3939
assert.equal(subagent.origin, "provider_native");
4040
assert.equal(subagent.createdBy, "agent");
4141
assert.equal(subagent.driver, "codex");
42-
assert.equal(subagent.title, "/root/hello_agent");
42+
assert.equal(subagent.title, "/root/hello");
4343
assert.equal(subagent.prompt, "");
4444
assert.equal(subagent.status, "completed");
4545
assert.equal(subagent.result, "Hello.");

0 commit comments

Comments
 (0)