Skip to content

Commit a22c7b1

Browse files
committed
🐛 Make plan approval and exhaustion mutually exclusive (#260)
A <Return> selects the value a value body publishes; it does not end that body, and a later failure wins over a value already selected. The approve branch returned the candidate and broke out of the review loop, then execution left the Session and reached the unconditional exhaustion <Fail>. Every approval settled as the ten-draft failure and no approved candidate could reach the host. Approval now binds the exact candidate to a null-sentinel result, and one exhaustive branch after the Session either returns it or fails. Only the arm taken runs, so an approved candidate has no later sibling failure to overtake it. Bounds, prompt wording, diagnostics, presentation, Session structure and the authored abort remain as they were. The review schema also needed an explicit string type: Ajv strict mode refuses minLength on an untyped property, so every review was refused before a person saw anything. Diagnostics reach their code block through <Json> rather than a JSON.stringify call in an expression, so serializing them is a document construct like everything else this policy does. The regression runs the packaged document itself, resolved against first-party declarations with an empty include list, against a scripted agent turn, a scripted approval and a test validator. It fails with the exhaustion message before this change. Claude-Session: https://claude.ai/code/session_015HcqB9kJM9KFnMNToAuZF6
1 parent ddd2999 commit a22c7b1

2 files changed

Lines changed: 201 additions & 4 deletions

File tree

‎packages/cli/src/documents/prompt-plan.md‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ turn belongs to. Nothing here runs the document being written. A candidate is
2424
text until a person approves it and the command runs it separately.
2525

2626
<Let as="round" value={0} />
27+
<Let as="approvedCandidate" value={null} />
2728

2829
<Session name={props.session}>
2930

@@ -62,7 +63,8 @@ found. A revision the person asked for is a new draft, so it gets its own three.
6263
<Prompt as="candidate">
6364
That document does not validate. These are the exact findings:
6465

65-
<CodeBlock value={JSON.stringify(assessment.diagnostics, null, 2)} language="json" />
66+
<Let as="findings"><Json value={assessment.diagnostics} /></Let>
67+
<CodeBlock value={findings} language="json" />
6668

6769
Return one complete replacement root document that resolves every finding above.
6870
Source only, no enclosing fence, no explanation. Keep the narrative that explains
@@ -93,7 +95,7 @@ into, so revision is not offered.
9395
required: ["decision"],
9496
additionalProperties: false,
9597
if: { properties: { decision: { const: "revise" } }, required: ["decision"] },
96-
then: { required: ["feedback"], properties: { feedback: { minLength: 1 } } },
98+
then: { required: ["feedback"], properties: { feedback: { type: "string", minLength: 1 } } },
9799
}}
98100
>
99101
This document was written for: {props.request}
@@ -104,12 +106,13 @@ This document was written for: {props.request}
104106
It still does not validate after three attempts to repair it. These are the
105107
remaining findings:
106108

107-
<CodeBlock value={JSON.stringify(assessment.diagnostics, null, 2)} language="json" />
109+
<Let as="findings"><Json value={assessment.diagnostics} /></Let>
110+
<CodeBlock value={findings} language="json" />
108111
</If>
109112
</Elicit>
110113

111114
<If condition={review.decision === "approve"}>
112-
<Return value={candidate} />
115+
<Let as="approvedCandidate" value={candidate} />
113116
<Break />
114117
</If>
115118

@@ -129,4 +132,14 @@ no explanation. Keep the narrative that explains what the document is for.
129132

130133
</Session>
131134

135+
A `<Return>` selects the value this document publishes; it does not end the
136+
document. So approval and exhaustion are written as the two arms of one branch,
137+
and only the arm taken runs — an approved candidate is returned with no later
138+
failure to overtake it, and a review that approved nothing fails instead.
139+
140+
<If condition={approvedCandidate !== null}>
141+
<Return value={approvedCandidate} />
142+
<Else>
132143
<Fail message="xmd prompt: ten drafts were reviewed and none was approved, so nothing was saved and nothing ran. Try again with a more specific request." />
144+
</Else>
145+
</If>
Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
/**
2+
* The packaged plan program, executed as itself.
3+
*
4+
* This runs the exact Markdown the CLI ships — read through the packaged
5+
* loader, not copied into a fixture — so what it proves is what a release does.
6+
* The seams around it are deterministic: a scriptable ACP runtime for the one
7+
* Agent turn, a scripted Elicitation answer for the review, and a test-only
8+
* validator standing in for the host capability that does not exist yet.
9+
*
10+
* The include list is empty on purpose. Repository component search must not be
11+
* able to supply `Loop`, `If`, `Return`, `Fail`, `CodeBlock` or the validator:
12+
* the policy under test is the one in the packaged document, resolved against
13+
* first-party declarations only.
14+
*/
15+
import { describe, it } from "@executablemd/test-support/bdd";
16+
import { expect } from "@executablemd/test-support/expect";
17+
import { scoped } from "effection";
18+
import type { Operation } from "effection";
19+
import {
20+
agentIdentityComponents,
21+
collect,
22+
Elicitation,
23+
installAgentComponents,
24+
registerAgentProvider,
25+
retainedSource,
26+
} from "@executablemd/core";
27+
import type { ElicitationRequest, Json } from "@executablemd/core";
28+
import { executeInstalled } from "@executablemd/core/host";
29+
import { InMemoryStream } from "@executablemd/durable-streams";
30+
import { createAcpxProvider } from "@executablemd/acp";
31+
32+
import { PROMPT_PLAN, readPackagedDocument } from "../src/packaged-document.ts";
33+
import { AGENT, useWorkingDirectory } from "./support/prompt-harness.ts";
34+
import { createFakeAcp, makeRegistry, makeStore } from "./support/fake-acp.ts";
35+
36+
/**
37+
* A candidate whose exact bytes are worth preserving.
38+
*
39+
* Leading and trailing blank lines, interior indentation, and a five-backtick
40+
* run: enough that trimming, re-fencing, or reconstructing it through the
41+
* presentation would all be visible in the assertion.
42+
*/
43+
const CANDIDATE = [
44+
"",
45+
"---",
46+
"returns: { type: string }",
47+
"---",
48+
"",
49+
"# A greeting",
50+
"",
51+
"This document explains itself before it does anything.",
52+
"",
53+
"`````markdown",
54+
" <Return value={`nested fence`} />",
55+
"`````",
56+
"",
57+
'<Return value="hello" />',
58+
"",
59+
].join("\n");
60+
61+
/** What the plan asked the validator about, in order. */
62+
interface PlanRun {
63+
validated: string[];
64+
reviews: ElicitationRequest[];
65+
prompts: string[];
66+
value: Json | undefined;
67+
failure: string | undefined;
68+
}
69+
70+
function* runPlan(): Operation<PlanRun> {
71+
const source = yield* readPackagedDocument(PROMPT_PLAN);
72+
const fake = createFakeAcp();
73+
fake.script({ reply: CANDIDATE });
74+
75+
const validated: string[] = [];
76+
const reviews: ElicitationRequest[] = [];
77+
let value: Json | undefined;
78+
let failure: string | undefined;
79+
80+
yield* scoped(function* () {
81+
const acpx = createAcpxProvider({
82+
createRuntime: fake.create,
83+
sessionStore: makeStore(),
84+
agentRegistry: makeRegistry({ [AGENT]: `${AGENT}-cmd` }),
85+
});
86+
yield* registerAgentProvider("acpx", acpx);
87+
yield* installAgentComponents({
88+
defaultAgent: AGENT,
89+
permissionMode: "deny-all",
90+
rootProvider: { factory: acpx, options: { defaultAgent: AGENT, permissionMode: "deny-all" } },
91+
});
92+
93+
yield* Elicitation.around(
94+
{
95+
// deno-lint-ignore require-yield
96+
*elicit([request], _next) {
97+
reviews.push(request);
98+
return { decision: "approve" };
99+
},
100+
},
101+
{ at: "min" },
102+
);
103+
104+
try {
105+
value = yield* collect(
106+
yield* executeInstalled(
107+
{
108+
...retainedSource("<prompt-plan>", source),
109+
stream: new InMemoryStream(),
110+
includes: [],
111+
props: {
112+
request: "ask me for my age and write the result to a file",
113+
syntax: "## Built-in components\n\n### `<File>`\n",
114+
session: "plan-regression",
115+
},
116+
},
117+
[
118+
{
119+
components: [
120+
...agentIdentityComponents(),
121+
{
122+
name: "ValidateCandidate",
123+
origin: "test",
124+
forms: ["self-closing"] as const,
125+
props: {
126+
type: "object",
127+
properties: { source: { type: "string" } },
128+
required: ["source"],
129+
additionalProperties: false,
130+
},
131+
returns: {
132+
type: "object",
133+
properties: {
134+
valid: { type: "boolean" },
135+
diagnostics: { type: "object" },
136+
},
137+
required: ["valid", "diagnostics"],
138+
additionalProperties: false,
139+
},
140+
// The deterministic seam only. It records what it was asked
141+
// about and says yes; the production validator is later work.
142+
factory: () =>
143+
// deno-lint-ignore require-yield
144+
function* validateCandidate(props: Record<string, Json>) {
145+
validated.push(String(props.source));
146+
return { valid: true, diagnostics: {} };
147+
},
148+
},
149+
],
150+
},
151+
],
152+
),
153+
);
154+
} catch (error) {
155+
failure = error instanceof Error ? error.message : String(error);
156+
}
157+
});
158+
159+
return { validated, reviews, prompts: fake.prompts, value, failure };
160+
}
161+
162+
describe("the packaged plan program", () => {
163+
it("returns the approved candidate's exact bytes and never reaches exhaustion", function* () {
164+
const run = yield* useWorkingDirectory(function* () {
165+
return yield* runPlan();
166+
});
167+
168+
// Approving the first valid candidate is one turn and one question. A
169+
// repair or revision turn here would mean the loops ran when they had
170+
// nothing to fix.
171+
expect(run.prompts).toHaveLength(1);
172+
expect(run.reviews).toHaveLength(1);
173+
174+
// The validator saw the Agent's complete close value, once, unaltered.
175+
expect(run.validated).toEqual([CANDIDATE]);
176+
177+
// The plan settled with a value rather than an authored failure. Before the
178+
// control-flow correction this was the ten-draft exhaustion message: a
179+
// `<Return>` selects a value but does not end the body, so the unconditional
180+
// `<Fail>` after the Session ran and won over every approval.
181+
expect(run.failure).toBe(undefined);
182+
expect(run.value).toBe(CANDIDATE);
183+
});
184+
});

0 commit comments

Comments
 (0)