Skip to content

Commit 7c131e2

Browse files
committed
🧪 Prove the shipped planning-document logic against stub agents
Six of #290's seven criteria now execute. `InstructionFiles` reaches Discovery's prompt and both of Planning's with the instruction paths and contents constrained literally, so a prompt that dropped or summarized them fails. A passing verdict returns true/true and sends no revision. A failing verdict sends its exact review, revision prompt and complete checkpoint record back to the implementor's own session, as that session's second turn and before the revised plan. Both bounded repair loops stop where the documents say: an initial malformed reply and two malformed corrections reach the final `<Parse>`, which is what the assertion pins — the third reply is the one quoted in the failure, so no third correction was sent. A material choice reaches the checkpoint, takes the `<Elicit>` path, and an explicit decline stays `proceed: false`; no material choice returns the documented continue record word for word. The seventh is not proven, and the document says so where the coverage would be rather than only here. Exercising all five failing iterations costs twenty agent turns in one invocation — as many as every other test combined, which take thirty-one seconds across seven timeboxes. A test's timeout is a fixed twenty seconds with no prop that raises it. The scenarios for the run are written and correct; what is missing is a way to run them. Lowering `max` would prove a different document than the one the workflow ships.
1 parent 1f56fdb commit 7c131e2

14 files changed

Lines changed: 393 additions & 0 deletions
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
<WhenPrompt as="a" template="{?a}Determine whether the user must be involved to resolve the plan review.{?b}Material to assess:{?c}MATERIAL-UNDER-REVIEW the plan widens the public route surface.{?d}Do not choose for the user.{?e}" />
2+
3+
{"requiresUser": true, "assessment": "ASSESSED-MATERIAL the plan changes the public route surface.", "question": "QUESTION-SCOPE Should the health route be public?", "options": ["public", "internal only"], "recommendation": "RECOMMEND internal only."}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
<WhenPrompt as="assessment" template="{?a}Determine whether the user must be involved to resolve the plan review.{?b}" />
2+
3+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the verdict passed and the plan is unchanged.", "question": "", "options": [], "recommendation": "Continue."}
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<WhenPrompt as="a1" template="{?a}Material to assess:{?b}PLAN-V1{?c}REVIEW-FAIL the plan cites no evidence for the mount point.{?d}Revision prompt:{?e}REVISE-EXACTLY cite the file and line that exposes the mount point.{?f}" />
2+
3+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the verdict passed and the plan is unchanged.", "question": "", "options": [], "recommendation": "Continue."}
4+
5+
<WhenPrompt as="a2" template="{?a}Material to assess:{?b}PLAN-V2{?c}REVIEW-PASS the revised plan cites router.ts line 40.{?d}" />
6+
7+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the verdict passed and the plan is unchanged.", "question": "", "options": [], "recommendation": "Continue."}
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
<WhenPrompt as="a1" template="{?a}Material to assess:{?b}PLAN-ROUND-1{?c}REVIEW-FAIL-1{?d}" />
2+
3+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the user keeps approving another round.", "question": "", "options": [], "recommendation": "Continue."}
4+
5+
<WhenPrompt as="a2" template="{?a}Material to assess:{?b}PLAN-ROUND-2{?c}REVIEW-FAIL-2{?d}" />
6+
7+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the user keeps approving another round.", "question": "", "options": [], "recommendation": "Continue."}
8+
9+
<WhenPrompt as="a3" template="{?a}Material to assess:{?b}PLAN-ROUND-3{?c}REVIEW-FAIL-3{?d}" />
10+
11+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the user keeps approving another round.", "question": "", "options": [], "recommendation": "Continue."}
12+
13+
<WhenPrompt as="a4" template="{?a}Material to assess:{?b}PLAN-ROUND-4{?c}REVIEW-FAIL-4{?d}" />
14+
15+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the user keeps approving another round.", "question": "", "options": [], "recommendation": "Continue."}
16+
17+
<WhenPrompt as="a5" template="{?a}Material to assess:{?b}PLAN-ROUND-5{?c}REVIEW-FAIL-5{?d}" />
18+
19+
{"requiresUser": false, "assessment": "ASSESSED-NO-CHOICE the user keeps approving another round.", "question": "", "options": [], "recommendation": "Continue."}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
<WhenPrompt as="plan1" template="{?a}Investigate the registered checkout.{?b}" />
2+
3+
PLAN-ROUND-1
4+
5+
A plan the planner will reject.
6+
7+
<WhenPrompt as="revision1" template="{?a}Revise the implementation plan using this review:{?b}REVIEW-FAIL-1{?c}REVISE-1{?d}" />
8+
9+
Acknowledged round 1.
10+
11+
<WhenPrompt as="plan2" template="{?a}Investigate the registered checkout.{?b}" />
12+
13+
PLAN-ROUND-2
14+
15+
A plan the planner will reject.
16+
17+
<WhenPrompt as="revision2" template="{?a}Revise the implementation plan using this review:{?b}REVIEW-FAIL-2{?c}REVISE-2{?d}" />
18+
19+
Acknowledged round 2.
20+
21+
<WhenPrompt as="plan3" template="{?a}Investigate the registered checkout.{?b}" />
22+
23+
PLAN-ROUND-3
24+
25+
A plan the planner will reject.
26+
27+
<WhenPrompt as="revision3" template="{?a}Revise the implementation plan using this review:{?b}REVIEW-FAIL-3{?c}REVISE-3{?d}" />
28+
29+
Acknowledged round 3.
30+
31+
<WhenPrompt as="plan4" template="{?a}Investigate the registered checkout.{?b}" />
32+
33+
PLAN-ROUND-4
34+
35+
A plan the planner will reject.
36+
37+
<WhenPrompt as="revision4" template="{?a}Revise the implementation plan using this review:{?b}REVIEW-FAIL-4{?c}REVISE-4{?d}" />
38+
39+
Acknowledged round 4.
40+
41+
<WhenPrompt as="plan5" template="{?a}Investigate the registered checkout.{?b}" />
42+
43+
PLAN-ROUND-5
44+
45+
A plan the planner will reject.
46+
47+
<WhenPrompt as="revision5" template="{?a}Revise the implementation plan using this review:{?b}REVIEW-FAIL-5{?c}REVISE-5{?d}" />
48+
49+
Acknowledged round 5.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
<WhenPrompt as="v1" template="{?a}PLAN-ROUND-1{?b}Review the plan against the handoff{?c}" />
2+
3+
{"passed": false, "review": "REVIEW-FAIL-1 still no evidence.", "revisionPrompt": "REVISE-1 cite the mount point."}
4+
5+
<WhenPrompt as="v2" template="{?a}PLAN-ROUND-2{?b}Review the plan against the handoff{?c}" />
6+
7+
{"passed": false, "review": "REVIEW-FAIL-2 still no evidence.", "revisionPrompt": "REVISE-2 cite the mount point."}
8+
9+
<WhenPrompt as="v3" template="{?a}PLAN-ROUND-3{?b}Review the plan against the handoff{?c}" />
10+
11+
{"passed": false, "review": "REVIEW-FAIL-3 still no evidence.", "revisionPrompt": "REVISE-3 cite the mount point."}
12+
13+
<WhenPrompt as="v4" template="{?a}PLAN-ROUND-4{?b}Review the plan against the handoff{?c}" />
14+
15+
{"passed": false, "review": "REVIEW-FAIL-4 still no evidence.", "revisionPrompt": "REVISE-4 cite the mount point."}
16+
17+
<WhenPrompt as="v5" template="{?a}PLAN-ROUND-5{?b}Review the plan against the handoff{?c}" />
18+
19+
{"passed": false, "review": "REVIEW-FAIL-5 still no evidence.", "revisionPrompt": "REVISE-5 cite the mount point."}
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<WhenPrompt as="a" template="{?a}Determine whether the user must be involved to{?b}" />
2+
3+
not json at all
4+
5+
<WhenPrompt as="c1" template="{?a}Correct your previous response without changing its meaning.{?b}" />
6+
7+
still not json
8+
9+
<WhenPrompt as="c2" template="{?a}Correct your previous response without changing its meaning.{?b}" />
10+
11+
third malformed reply
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
<WhenPrompt as="plan" template="{?a}Investigate the registered checkout.{?b}" />
2+
3+
PLAN-V1
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<WhenPrompt as="v" template="{?a}Review the plan against the handoff{?b}" />
2+
3+
not json at all
4+
5+
<WhenPrompt as="c1" template="{?a}Correct your previous response without changing its meaning.{?b}" />
6+
7+
still not json
8+
9+
<WhenPrompt as="c2" template="{?a}Correct your previous response without changing its meaning.{?b}" />
10+
11+
third malformed reply
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<WhenPrompt
2+
as="plan"
3+
template="{?a}Root instructions: prefer evidence over assertion.{?b}Nested instructions: never edit a test to make it pass.{?c}HANDOFF{?d}Investigate the registered checkout.{?e}"
4+
/>
5+
6+
PLAN-V1
7+
8+
Add `/health` to the existing router mount point. Evidence: the router already
9+
exposes one. Validation: a route test. No environmental effects.

0 commit comments

Comments
 (0)