Skip to content

Commit 055c364

Browse files
committed
🧪 Run the exhausted planning loop where twenty agent turns fit
#290's last criterion needs five failing iterations of `Planning` in one invocation: five plan prompts, five verdicts, five checkpoint assessments and the five revisions a failing round sends. Twenty ACP round trips do not fit a test's fixed twenty-second timeout, and lowering `max` would prove a different document than the one the workflow ships. So the run moves to an integration test that keeps the document and replaces the transport. The real `Planning` resolves from the workflow directory on the same search path `xmd test` derives, under one execution and one contextual working directory, against a synchronous stub root Agent provider installed through the public `rootProvider` seam. It finishes in about half a second. What it proves is the ordering as much as the outcome: exactly twenty calls, five of each turn in the order the document writes them, each routed to the session the document sends it to, every verdict validly failing and every checkpoint explicitly continuing, and each round's review, revision prompt and assessment carried into that round's revision. The returned pair is uniquely exhausted, and the plan, review and revision prompt are round five's. The root writes `start.md`'s own gate with a later effect behind it. That prompt never runs, which is what the twentieth-and-final call count says. Removing `verdictPassed` from the gate makes the count twenty-one and fails the test, so the assertion discriminates the gate rather than restating it.
1 parent 5f93d8d commit 055c364

6 files changed

Lines changed: 293 additions & 114 deletions

File tree

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
/**
2+
* #290 criterion 6 — an exhausted planning loop.
3+
*
4+
* `Planning` declares `<Loop name="planning" max={5}>`, so exercising every
5+
* failing iteration costs twenty agent turns in one invocation. That does not
6+
* fit the fixed twenty-second `<Test>` timeout when each turn is an ACP round
7+
* trip, so the run happens here instead: the real `Planning` component, one
8+
* execution, one contextual working directory, and a synchronous stub root
9+
* Agent provider in place of the transport.
10+
*
11+
* Nothing about the document under test changes. The component resolves from
12+
* the workflow directory exactly as `xmd test` resolves it, including the
13+
* test-local `Agent.AddDir` stub, and the loop bound is the one the workflow
14+
* ships.
15+
*
16+
* What `start.md` does with the returned pair — placing `<Implementation>`
17+
* behind the same gate and reporting exhaustion as awaiting direction inside
18+
* `<Dir>` and the root `<Output>` — stays the composition probe's evidence.
19+
* This test proves the pair, the turns that produce it, and that a later effect
20+
* written behind that gate does not run.
21+
*/
22+
23+
import { describe, it } from "@executablemd/test-support/bdd";
24+
import { expect } from "@executablemd/test-support/expect";
25+
import { ensure, scoped } from "effection";
26+
import type { Operation, Stream } from "effection";
27+
import { InMemoryStream } from "@executablemd/durable-streams";
28+
import { ensureDir, rm, writeTextFile } from "@effectionx/fs";
29+
import { randomUUID } from "node:crypto";
30+
import * as path from "node:path";
31+
import * as os from "node:os";
32+
import * as url from "node:url";
33+
import { execute, installAgentComponents, Agent } from "@executablemd/core";
34+
import type {
35+
AgentPromptEvent,
36+
AgentProviderFactory,
37+
AgentProviderOptions,
38+
Json,
39+
PromptOptions,
40+
Session,
41+
} from "@executablemd/core";
42+
43+
const HERE = path.dirname(url.fileURLToPath(import.meta.url));
44+
const WORKFLOW = path.resolve(HERE, "../../workflows/adversarial-implementation");
45+
/** The same search path `xmd test` derives for a document in `tests/`. */
46+
const COMPONENT_DIRS = [path.join(WORKFLOW, "tests"), WORKFLOW];
47+
48+
const MAX_ITERATIONS = 5;
49+
50+
/** Which prompt a turn is, read from the text the document actually sent. */
51+
type Turn = "plan" | "verdict" | "assessment" | "revision" | "sentinel";
52+
53+
function classify(content: string): Turn {
54+
if (content.includes("Revise the implementation plan using this review:")) {
55+
return "revision";
56+
}
57+
if (content.includes("Investigate the registered checkout.")) {
58+
return "plan";
59+
}
60+
if (content.includes("Review the plan against the handoff")) {
61+
return "verdict";
62+
}
63+
if (content.includes("Determine whether the user must be involved to")) {
64+
return "assessment";
65+
}
66+
return "sentinel";
67+
}
68+
69+
interface Call {
70+
readonly turn: Turn;
71+
readonly session: string;
72+
readonly content: string;
73+
}
74+
75+
function sessionName(options: PromptOptions | undefined): string {
76+
const session = options?.session;
77+
if (typeof session === "object" && session !== null && "sessionKey" in session) {
78+
return String((session as Session).sessionKey);
79+
}
80+
return typeof session === "string" ? session : "<none>";
81+
}
82+
83+
/**
84+
* A verdict that fails validly, and an assessment that explicitly continues.
85+
* Neither is malformed: a repair loop and an exhausted planning loop are
86+
* different bounds, and this run exercises only the second.
87+
*/
88+
function reply(turn: Turn, round: number): string {
89+
switch (turn) {
90+
case "plan":
91+
return `PLAN-ROUND-${round}`;
92+
case "verdict":
93+
return JSON.stringify({
94+
passed: false,
95+
review: `REVIEW-FAIL-${round}`,
96+
revisionPrompt: `REVISE-${round}`,
97+
});
98+
case "assessment":
99+
return JSON.stringify({
100+
requiresUser: false,
101+
assessment: `ASSESSED-${round}`,
102+
question: "",
103+
options: [],
104+
recommendation: "Continue.",
105+
});
106+
case "revision":
107+
return `ACKNOWLEDGED-${round}`;
108+
case "sentinel":
109+
return "SENTINEL-RAN";
110+
}
111+
}
112+
113+
function stubProvider(calls: Call[]): AgentProviderFactory {
114+
const rounds = new Map<Turn, number>();
115+
return function* (options) {
116+
yield* Agent.around(
117+
{
118+
// deno-lint-ignore require-yield
119+
*agent([name]) {
120+
return name ?? options.defaultAgent;
121+
},
122+
// deno-lint-ignore require-yield
123+
*session([name]) {
124+
return { sessionKey: `${name ?? "default"}`, cwd: "/" };
125+
},
126+
// deno-lint-ignore require-yield
127+
*prompt([content, promptOptions]) {
128+
const turn = classify(content);
129+
const round = (rounds.get(turn) ?? 0) + 1;
130+
rounds.set(turn, round);
131+
calls.push({ turn, session: sessionName(promptOptions), content });
132+
return stream(reply(turn, round), promptOptions);
133+
},
134+
},
135+
{ at: "min" },
136+
);
137+
};
138+
}
139+
140+
function stream(text: string, options: PromptOptions | undefined): Stream<AgentPromptEvent, string> {
141+
return {
142+
*[Symbol.iterator]() {
143+
const session: Session = typeof options?.session === "object"
144+
? options.session
145+
: { sessionKey: "default", cwd: "/" };
146+
const events: AgentPromptEvent[] = [
147+
{ type: "started", agent: options?.agent ?? "stub", session },
148+
{ type: "text_delta", text },
149+
{ type: "terminal", status: "completed" },
150+
];
151+
let index = 0;
152+
return {
153+
// deno-lint-ignore require-yield
154+
*next() {
155+
if (index < events.length) {
156+
return { done: false, value: events[index++]! };
157+
}
158+
return { done: true, value: text };
159+
},
160+
};
161+
},
162+
};
163+
}
164+
165+
/**
166+
* The root invokes the real `Planning`, writes `start.md`'s own gate, and puts
167+
* a later effect behind it. A gate that admitted an exhausted plan would send
168+
* that prompt, and the recorder would see a twenty-first call.
169+
*/
170+
const ROOT = `---
171+
returns:
172+
proceed: { type: boolean }
173+
verdictPassed: { type: boolean }
174+
plan: { type: string }
175+
review: { type: string }
176+
revisionPrompt: { type: string }
177+
---
178+
179+
<Planning
180+
handoff="HANDOFF the route is /health."
181+
handoffCheckpoint={{proceed: true, assessment: "HANDOFF-ASSESSED", response: "approved", rationale: "clear"}}
182+
instructions="INSTRUCTIONS"
183+
planner="planner"
184+
implementor="implementor"
185+
worktree="."
186+
as="planning"
187+
/>
188+
189+
<If condition={planning.decision.proceed && planning.verdictPassed}>
190+
<Prompt agent="sentinel" text="A later durable effect behind the production gate." />
191+
</If>
192+
193+
<Return value={{
194+
proceed: planning.decision.proceed,
195+
verdictPassed: planning.verdictPassed,
196+
plan: planning.plan,
197+
review: planning.review,
198+
revisionPrompt: planning.revisionPrompt
199+
}} />
200+
`;
201+
202+
function* runPlanning(): Operation<{ value: Json; calls: Call[] }> {
203+
const calls: Call[] = [];
204+
const dir = path.join(os.tmpdir(), `xmd-290-${randomUUID()}`);
205+
yield* ensureDir(dir);
206+
return yield* scoped(function* () {
207+
yield* ensure(() => rm(dir, { recursive: true, force: true }));
208+
const options: AgentProviderOptions = { defaultAgent: "stub", permissionMode: "deny-all" };
209+
yield* installAgentComponents({ rootProvider: { factory: stubProvider(calls), options } });
210+
211+
const root = path.join(dir, "exhaustion-root.md");
212+
yield* writeTextFile(root, ROOT);
213+
const execution = yield* execute({
214+
path: root,
215+
stream: new InMemoryStream(),
216+
componentDirs: COMPONENT_DIRS,
217+
});
218+
const subscription = yield* execution.output;
219+
let next = yield* subscription.next();
220+
while (!next.done) {
221+
next = yield* subscription.next();
222+
}
223+
const result = yield* execution;
224+
if (!result.ok) {
225+
throw result.error;
226+
}
227+
return { value: result.value, calls };
228+
});
229+
}
230+
231+
describe("#290 criterion 6 — an exhausted planning loop", () => {
232+
it("runs five failing iterations and returns the exhausted pair without authorizing anything", function* () {
233+
const { value, calls } = yield* runPlanning();
234+
const returned = value as Record<string, Json>;
235+
236+
// Exactly twenty calls: no sixth iteration, and the sentinel never ran.
237+
expect(calls).toHaveLength(MAX_ITERATIONS * 4);
238+
expect(calls.filter((call) => call.turn === "sentinel")).toHaveLength(0);
239+
240+
// Five of each turn, in the order the document writes them.
241+
const expected = Array.from({ length: MAX_ITERATIONS }).flatMap(() => [
242+
"plan",
243+
"verdict",
244+
"assessment",
245+
"revision",
246+
]);
247+
expect(calls.map((call) => call.turn)).toEqual(expected);
248+
249+
// Each turn reaches the session the document routes it to, and the
250+
// revision goes back to the implementor's own conversation.
251+
const sessions = new Map(calls.map((call) => [call.turn, call.session]));
252+
expect(sessions.get("plan")).toBe("implementor");
253+
expect(sessions.get("revision")).toBe("implementor");
254+
expect(sessions.get("verdict")).toBe("planner");
255+
expect(sessions.get("assessment")).toBe("user-checkpoint");
256+
257+
// The outcome is uniquely exhausted. Convergence would be true/true and a
258+
// decline would be false; nothing else produces this pair.
259+
expect(returned["proceed"]).toBe(true);
260+
expect(returned["verdictPassed"]).toBe(false);
261+
262+
// What the stage returns is round five's, not an earlier round's.
263+
expect(returned["plan"]).toContain(`PLAN-ROUND-${MAX_ITERATIONS}`);
264+
expect(returned["review"]).toBe(`REVIEW-FAIL-${MAX_ITERATIONS}`);
265+
expect(returned["revisionPrompt"]).toBe(`REVISE-${MAX_ITERATIONS}`);
266+
});
267+
268+
it("carries each round's verdict into that round's revision prompt", function* () {
269+
const { calls } = yield* runPlanning();
270+
const revisions = calls.filter((call) => call.turn === "revision");
271+
expect(revisions).toHaveLength(MAX_ITERATIONS);
272+
revisions.forEach((revision, index) => {
273+
const round = index + 1;
274+
expect(revision.content).toContain(`REVIEW-FAIL-${round}`);
275+
expect(revision.content).toContain(`REVISE-${round}`);
276+
expect(revision.content).toContain(`ASSESSED-${round}`);
277+
});
278+
});
279+
});

‎workflows/adversarial-implementation/primitives.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -410,7 +410,7 @@ replay expands the same fragment without asking the Agent again.
410410
| --- | --- | --- |
411411
| namespaced document props | #305 | shipped |
412412
| synchronize this living target with settled contracts | #292 | this change |
413-
| prove the shipped planning-document logic | #290 | open |
413+
| prove the shipped planning-document logic | #290 | this change — the proof lands here; #290 closes on delivery |
414414
| compose the supervised workflow, and make its stage components reachable under `start` and `resume` | #301 | open |
415415
| resume from explicit durable signals | #300 | open |
416416
| default-on secret rejection before journal persistence | #199 | open — guard and scanner built |

‎workflows/adversarial-implementation/tests/agents/exhaustion-checkpoint.md‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎workflows/adversarial-implementation/tests/agents/exhaustion-implementor.md‎

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎workflows/adversarial-implementation/tests/agents/exhaustion-planner.md‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

0 commit comments

Comments
 (0)