Skip to content

Commit 4fd53a6

Browse files
authored
Merge pull request #179 from aictrl-dev/fix/issue-135-schema-reject-corrective-turn
fix(run): give schema runs a corrective turn after a permission rejection (#135)
2 parents 842eeb0 + 3fd36ac commit 4fd53a6

5 files changed

Lines changed: 196 additions & 40 deletions

File tree

‎EVENTS.md‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -583,8 +583,10 @@ value in the terminal event below.
583583

584584
### `structured_output_rejected`
585585

586-
Emitted once per rejected StructuredOutput attempt, including an unusable result
587-
file and a model turn that finishes in prose without a final tool call:
586+
Emitted once per rejected attempt against the corrective budget: a rejected
587+
StructuredOutput call (including an unusable result file), a model turn that
588+
finishes in prose without a final tool call, or a step stopped by a rejected
589+
tool call (such as a headless permission rejection):
588590

589591
```json
590592
{
@@ -622,6 +624,17 @@ asks it to fix the file at the JSON-encoded path. Malformed tool arguments (not
622624
A prose-only finish emits the fixed diagnostic
623625
`{ "path": "", "keyword": "missing", "message": "write the final result as JSON to a file in the working directory, then call StructuredOutput with its path" }`
624626
and receives an ephemeral reminder before the next turn while budget remains.
627+
A step that stops on a rejected tool call is handled the same way. In headless
628+
mode this is a permission auto-rejected because there is no user to approve it
629+
(see `permission_rejected`); through the SDK it can also be a dismissed
630+
question. It emits the fixed diagnostic
631+
`{ "path": "", "keyword": "rejected", "message": "a tool call was rejected; write the final result as JSON to a file in the working directory, then call StructuredOutput with its path" }`.
632+
The reminder names the rejected tools, not their arguments, and asks the model to
633+
finish with what it already has. If the same step contains a rejected
634+
StructuredOutput call, that call is the step's attempt: no `rejected` diagnostic
635+
is added, but the corrective reminder still follows. Both corrective turns force
636+
the StructuredOutput tool choice. Without `--output-schema`, a rejected tool
637+
call still ends the run.
625638
Other tools remain usable until a valid final result; their outputs are not final
626639
results. Only the `{ path }` tool schema is sent to the provider; local validation
627640
of the file always uses the canonical schema.
@@ -642,19 +655,21 @@ or:
642655
```
643656

644657
`attempts` counts StructuredOutput calls, including rejected/unparseable ones,
645-
and prose-only finishes without a final call. Other tool calls do not count.
658+
prose-only finishes without a final call, and steps stopped by a rejected tool
659+
call without a StructuredOutput call. Other tool calls do not count.
646660
Compaction turns do not consume a missing-result attempt. Discarded provider
647661
streams restore the budget and captured result; their rejection events remain
648662
as telemetry. A valid call in a parallel step wins regardless of call order if
649663
that step started with budget remaining. The attempt count includes every call
650664
in that step and can exceed `maxAttempts`; the N+1 bound applies to corrective
651665
turns.
652666
When the budget is spent, the last rejection determines the reason: `missing`
653-
for a prose-only finish, otherwise `exhausted`. Failure reasons are:
667+
for a prose-only finish or a rejected tool call, otherwise `exhausted`.
668+
Failure reasons are:
654669

655670
- `exhausted`: invalid or unparseable calls spent the corrective budget.
656-
- `missing`: prose finishes spent the budget, or the session ended normally
657-
without an accepted result (including headless permission rejection).
671+
- `missing`: prose finishes or rejected tool calls spent the budget, or the
672+
session ended normally without an accepted result.
658673
- `step_limit`: the agent reached its configured step cap without a result.
659674
- `aborted`: the invocation was cancelled, including SIGINT/SIGTERM.
660675
- `error`: a provider or session error, including stream idle timeout or an
@@ -668,4 +683,5 @@ Exit codes are **0** on accepted results, **2** for schema configuration errors,
668683
**3** for exhausted attempts, missing output and step limits. Provider errors and
669684
timeouts keep **1**; SIGINT and SIGTERM keep **130** and **143**.
670685
`StructuredOutputError.data.retries` records the number of corrective
671-
attempts used (initial attempt excluded), including prose-only corrective turns.
686+
attempts used (initial attempt excluded), including corrective turns after a
687+
prose-only finish or a rejected tool call.

‎packages/cli/src/session/processor.ts‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ export namespace SessionProcessor {
3434
abort: AbortSignal
3535
}) {
3636
const toolcalls: Record<string, MessageV2.ToolPart> = {}
37-
let blocked = false
37+
// Tools whose rejected permission or dismissed question stopped the loop.
38+
const blocked = new Set<string>()
3839
let attempt = 0
3940
let needsCompaction = false
4041
const retried: MessageV2.ToolPart[] = []
@@ -56,6 +57,9 @@ export namespace SessionProcessor {
5657
get message() {
5758
return input.assistantMessage
5859
},
60+
get blocked() {
61+
return [...blocked]
62+
},
5963
partFromToolCall(toolCallID: string) {
6064
return toolcalls[toolCallID]
6165
},
@@ -285,10 +289,11 @@ export namespace SessionProcessor {
285289
})
286290

287291
if (
288-
value.error instanceof PermissionNext.RejectedError ||
289-
value.error instanceof Question.RejectedError
292+
shouldBreak &&
293+
(value.error instanceof PermissionNext.RejectedError ||
294+
value.error instanceof Question.RejectedError)
290295
) {
291-
blocked = shouldBreak
296+
blocked.add(match.tool)
292297
}
293298
delete toolcalls[value.toolCallId]
294299
}
@@ -450,6 +455,8 @@ export namespace SessionProcessor {
450455
const retry = SessionRetry.retryable(error)
451456
if (retry !== undefined) {
452457
reset?.()
458+
// The discarded attempt's rejections never reach the model's history.
459+
blocked.clear()
453460
retried.push(
454461
...(await MessageV2.parts(input.assistantMessage.id)).filter(
455462
(part): part is MessageV2.ToolPart => part.type === "tool" && delivered.has(part.id),
@@ -528,7 +535,7 @@ export namespace SessionProcessor {
528535
// status; headless consumers stop reading when they observe idle.
529536
await Session.updateMessage(input.assistantMessage)
530537
if (needsCompaction) return "compact"
531-
if (blocked) return "stop"
538+
if (blocked.size) return "stop"
532539
if (input.assistantMessage.error) return "stop"
533540
return "continue"
534541
}

‎packages/cli/src/session/prompt.ts‎

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,8 @@ IMPORTANT:
5858
const STRUCTURED_OUTPUT_MISSING =
5959
"write the final result as JSON to a file in the working directory, then call StructuredOutput with its path"
6060

61+
const STRUCTURED_OUTPUT_REJECTED = `a tool call was rejected; ${STRUCTURED_OUTPUT_MISSING}`
62+
6163
const STRUCTURED_OUTPUT_SYSTEM_PROMPT = `IMPORTANT: The user has requested structured output. Write your final result as JSON to a file in the working directory, then call the StructuredOutput tool with that file's path. Do NOT respond with plain text.`
6264

6365
export namespace SessionPrompt {
@@ -84,6 +86,23 @@ export namespace SessionPrompt {
8486
return !hasToolCalls(await load())
8587
}
8688

89+
// A schema turn that stops without a result earns a corrective reminder: a
90+
// prose-only finish, or a rejected tool call (a permission auto-rejected in
91+
// headless mode, or a dismissed question; #135). The reminder names the
92+
// rejected tools, never their input.
93+
async function missingResult(processor: SessionProcessor.Info) {
94+
if (processor.blocked.length && !processor.message.error)
95+
return {
96+
diagnostic: { path: "", keyword: "rejected", message: STRUCTURED_OUTPUT_REJECTED },
97+
reminder: `Tool call rejected: ${processor.blocked.join(", ")}. Using what you already have, ${STRUCTURED_OUTPUT_MISSING}`,
98+
}
99+
if (await missingStructuredOutput(processor.message, () => MessageV2.parts(processor.message.id)))
100+
return {
101+
diagnostic: { path: "", keyword: "missing", message: STRUCTURED_OUTPUT_MISSING },
102+
reminder: STRUCTURED_OUTPUT_MISSING,
103+
}
104+
}
105+
87106
const state = Instance.state(
88107
() => {
89108
const data: Record<
@@ -329,7 +348,7 @@ export namespace SessionPrompt {
329348
let structuredOutput: unknown | undefined
330349
let attempts = 0
331350
let rejected = 0
332-
let reminder = false
351+
let reminder: string | undefined
333352
let user: string | undefined
334353
let contract:
335354
| {
@@ -383,7 +402,7 @@ export namespace SessionPrompt {
383402
: undefined
384403
attempts = 0
385404
rejected = 0
386-
reminder = false
405+
reminder = undefined
387406
structuredOutput = undefined
388407
}
389408
const lastAssistantMsg = msgs.findLast((msg) => msg.info.id === lastAssistant?.id)
@@ -787,6 +806,7 @@ export namespace SessionPrompt {
787806
system.push(STRUCTURED_OUTPUT_SYSTEM_PROMPT)
788807
}
789808

809+
const prior = attempts
790810
const result = await processor.process({
791811
user: lastUser,
792812
agent,
@@ -799,7 +819,7 @@ export namespace SessionPrompt {
799819
? [
800820
{
801821
role: "user" as const,
802-
content: `<system-reminder>${STRUCTURED_OUTPUT_MISSING}</system-reminder>`,
822+
content: `<system-reminder>${reminder}</system-reminder>`,
803823
},
804824
]
805825
: []),
@@ -859,22 +879,22 @@ export namespace SessionPrompt {
859879
continue
860880
}
861881

862-
reminder =
863-
!!contract && (await missingStructuredOutput(processor.message, () => MessageV2.parts(processor.message.id)))
864-
if (reminder) {
865-
await rejection([{ path: "", keyword: "missing", message: STRUCTURED_OUTPUT_MISSING }])
866-
}
882+
const missing = contract ? await missingResult(processor) : undefined
883+
reminder = missing?.reminder
884+
// A rejected StructuredOutput call in the same step already spent its attempt.
885+
const missed = missing !== undefined && attempts === prior
886+
if (missed) await rejection([missing.diagnostic])
867887

868888
if (contract && !processor.message.error && rejected > contract.format.retryCount) {
869889
processor.message.error = new MessageV2.StructuredOutputError({
870-
message: reminder
890+
message: missed
871891
? "Model did not produce structured output"
872892
: "Structured output corrective attempts exhausted",
873893
retries: Math.max(0, attempts - 1),
874894
}).toObject()
875895
await Session.updateMessage(processor.message)
876896
await Bus.publish(Session.Event.Error, { sessionID, error: processor.message.error })
877-
outcome = { status: "failed", reason: reminder ? "missing" : "exhausted", attempts }
897+
outcome = { status: "failed", reason: missed ? "missing" : "exhausted", attempts }
878898
break
879899
}
880900

‎packages/cli/test/cli/run-output-schema.test.ts‎

Lines changed: 71 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ async function run(
2121
queued?: boolean
2222
attach?: boolean
2323
locked?: boolean
24-
denied?: boolean
2524
} = {},
2625
) {
2726
await using tmp = await tmpdir()
@@ -71,12 +70,14 @@ async function run(
7170
index: 0,
7271
id: `call_${requests.length}`,
7372
type: "function",
74-
function: options.denied
75-
? { name: "read", arguments: JSON.stringify({ filePath: "/outside-fixture.txt" }) }
76-
: {
77-
name: "StructuredOutput",
78-
arguments: pathArgs(tmp.path, args, `so_${requests.length}.json`),
79-
},
73+
// "denied" reads outside the run directory, which headless mode auto-rejects.
74+
function:
75+
args === "denied"
76+
? { name: "read", arguments: JSON.stringify({ filePath: "/outside-fixture.txt" }) }
77+
: {
78+
name: "StructuredOutput",
79+
arguments: pathArgs(tmp.path, args, `so_${requests.length}.json`),
80+
},
8081
},
8182
],
8283
},
@@ -85,7 +86,7 @@ async function run(
8586
],
8687
},
8788
{
88-
choices: [{ index: 0, delta: {}, finish_reason: options.denied ? "tool-calls" : "stop" }],
89+
choices: [{ index: 0, delta: {}, finish_reason: args === "denied" ? "tool-calls" : "stop" }],
8990
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
9091
},
9192
]
@@ -762,22 +763,78 @@ test("JSON mode without output-result carries the accepted value only in termina
762763
])
763764
}, 25000)
764765

765-
test("headless permission rejection reports missing output with no session error", async () => {
766+
const REJECTED_REMINDER =
767+
"<system-reminder>Tool call rejected: read. Using what you already have, write the final result as JSON to a file in the working directory, then call StructuredOutput with its path</system-reminder>"
768+
769+
test("headless permission rejection gets a corrective turn that can still succeed", async () => {
766770
const result = await run({
767-
denied: true,
771+
inputs: ["denied", '{"result":"accepted"}'],
772+
flags: ["--output-schema", "schema.json", "--output-result", "result.json"],
773+
})
774+
expect(result.exit, result.stderr + result.stdout).toBe(0)
775+
expect(result.requests).toHaveLength(2)
776+
expect(result.requests.map((request) => request.tool_choice)).toEqual([
777+
"required",
778+
{ type: "function", function: { name: "StructuredOutput" } },
779+
])
780+
const messages = result.requests[1].messages as { role: string; content: unknown }[]
781+
expect(messages.filter((message) => JSON.stringify(message.content).includes("Tool call rejected"))).toEqual([
782+
{ role: "user", content: REJECTED_REMINDER },
783+
])
784+
expect(result.result).toBe(JSON.stringify({ result: "accepted" }, null, 2) + "\n")
785+
expect(result.events.filter((event) => event.type === "permission_rejected")).toHaveLength(1)
786+
expect(result.events.filter((event) => event.type === "session_error")).toEqual([])
787+
expect(result.events.filter((event) => event.type === "structured_output_rejected")).toMatchObject([
788+
{
789+
attempt: 1,
790+
maxAttempts: 3,
791+
errors: [
792+
{
793+
path: "",
794+
keyword: "rejected",
795+
message:
796+
"a tool call was rejected; write the final result as JSON to a file in the working directory, then call StructuredOutput with its path",
797+
},
798+
],
799+
},
800+
])
801+
expect(result.events.filter((event) => event.type === "structured_output")).toMatchObject([
802+
{ status: "accepted", attempts: 2, value: { result: "accepted" } },
803+
])
804+
}, 25000)
805+
806+
test("repeated headless permission rejections spend the budget as missing output", async () => {
807+
const result = await run({
808+
inputs: ["denied"],
768809
existing: "untouched",
769810
flags: ["--output-schema", "schema.json", "--output-result", "result.json"],
770811
})
771812
expect(result.exit, result.stderr + result.stdout).toBe(3)
772-
expect(result.requests).toHaveLength(1)
813+
expect(result.requests).toHaveLength(3)
773814
expect(result.result).toBe("untouched")
774-
expect(result.events.filter((event) => event.type === "permission_rejected")).toHaveLength(1)
775-
expect(result.events.filter((event) => event.type === "session_error")).toEqual([])
815+
expect(result.events.filter((event) => event.type === "permission_rejected")).toHaveLength(3)
816+
expect(
817+
result.events
818+
.filter((event) => event.type === "structured_output_rejected")
819+
.map((event) => [event.attempt, (event.errors as { keyword: string }[])[0].keyword]),
820+
).toEqual([
821+
[1, "rejected"],
822+
[2, "rejected"],
823+
[3, "rejected"],
824+
])
776825
expect(result.events.filter((event) => event.type === "structured_output")).toMatchObject([
777-
{ status: "failed", reason: "missing", attempts: 0 },
826+
{ status: "failed", reason: "missing", attempts: 3 },
778827
])
779828
}, 25000)
780829

830+
test("plain run still stops after a headless permission rejection", async () => {
831+
const result = await run({ inputs: ["denied"], flags: [] })
832+
expect(result.exit, result.stderr + result.stdout).toBe(0)
833+
expect(result.requests).toHaveLength(1)
834+
expect(result.events.filter((event) => event.type === "permission_rejected")).toHaveLength(1)
835+
expect(result.events.filter((event) => String(event.type).startsWith("structured_output"))).toEqual([])
836+
}, 25000)
837+
781838
test("schema config NDJSON clips the specific compilation cause", async () => {
782839
const result = await run({ content: JSON.stringify({ type: "object", ["unknown".repeat(100)]: true }) })
783840
expect(result.exit).toBe(2)

0 commit comments

Comments
 (0)