Skip to content

Commit 716edb5

Browse files
byapparovclaude
andcommitted
fix(run): count one attempt per step and name rejected calls neutrally (#179 review)
- A step that rejects a StructuredOutput file and also stops on a rejected tool call already spent its attempt; do not count a second `rejected` diagnostic, but still send the corrective reminder. Exhaustion then reports `exhausted`, not `missing`. - Question.RejectedError also stops the loop, so word the reminder and diagnostic as a rejected tool call (keyword `rejected`) instead of a denied permission, and document both sources in EVENTS.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent df5ccb2 commit 716edb5

5 files changed

Lines changed: 65 additions & 38 deletions

File tree

‎EVENTS.md‎

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -562,7 +562,7 @@ value in the terminal event below.
562562

563563
Emitted once per rejected StructuredOutput attempt, including an unusable result
564564
file, a model turn that finishes in prose without a final tool call, and a step
565-
stopped by a headless permission rejection:
565+
stopped by a rejected tool call (such as a headless permission rejection):
566566

567567
```json
568568
{
@@ -600,14 +600,17 @@ asks it to fix the file at the JSON-encoded path. Malformed tool arguments (not
600600
A prose-only finish emits the fixed diagnostic
601601
`{ "path": "", "keyword": "missing", "message": "write the final result as JSON to a file in the working directory, then call StructuredOutput with its path" }`
602602
and receives an ephemeral reminder before the next turn while budget remains.
603-
A step that stops because a tool call's permission was auto-rejected (headless
604-
mode has no user to approve; see `permission_rejected`) is handled the same way.
605-
It emits the fixed diagnostic
606-
`{ "path": "", "keyword": "denied", "message": "a tool call was denied; write the final result as JSON to a file in the working directory, then call StructuredOutput with its path" }`.
607-
The reminder names the denied tools, not their arguments, and asks the model to
608-
finish with what it already has. Both corrective turns force the StructuredOutput
609-
tool choice. Without `--output-schema`, a permission rejection
610-
still ends the run.
603+
A step that stops on a rejected tool call is handled the same way. In headless
604+
mode this is a permission auto-rejected because there is no user to approve it
605+
(see `permission_rejected`); through the SDK it can also be a dismissed
606+
question. It emits the fixed diagnostic
607+
`{ "path": "", "keyword": "rejected", "message": "a tool call was rejected; write the final result as JSON to a file in the working directory, then call StructuredOutput with its path" }`.
608+
The reminder names the rejected tools, not their arguments, and asks the model to
609+
finish with what it already has. If the same step contains a rejected
610+
StructuredOutput call, that call is the step's attempt: no `rejected` diagnostic
611+
is added, but the corrective reminder still follows. Both corrective turns force
612+
the StructuredOutput tool choice. Without `--output-schema`, a rejected tool
613+
call still ends the run.
611614
Other tools remain usable until a valid final result; their outputs are not final
612615
results. Only the `{ path }` tool schema is sent to the provider; local validation
613616
of the file always uses the canonical schema.
@@ -628,20 +631,20 @@ or:
628631
```
629632

630633
`attempts` counts StructuredOutput calls, including rejected/unparseable ones,
631-
prose-only finishes without a final call, and steps stopped by a permission
632-
rejection. Other tool calls do not count.
634+
prose-only finishes without a final call, and steps stopped by a rejected tool
635+
call without a StructuredOutput call. Other tool calls do not count.
633636
Compaction turns do not consume a missing-result attempt. Discarded provider
634637
streams restore the budget and captured result; their rejection events remain
635638
as telemetry. A valid call in a parallel step wins regardless of call order if
636639
that step started with budget remaining. The attempt count includes every call
637640
in that step and can exceed `maxAttempts`; the N+1 bound applies to corrective
638641
turns.
639642
When the budget is spent, the last rejection determines the reason: `missing`
640-
for a prose-only finish or a permission rejection, otherwise `exhausted`.
643+
for a prose-only finish or a rejected tool call, otherwise `exhausted`.
641644
Failure reasons are:
642645

643646
- `exhausted`: invalid or unparseable calls spent the corrective budget.
644-
- `missing`: prose finishes or permission rejections spent the budget, or the
647+
- `missing`: prose finishes or rejected tool calls spent the budget, or the
645648
session ended normally without an accepted result.
646649
- `step_limit`: the agent reached its configured step cap without a result.
647650
- `aborted`: the invocation was cancelled, including SIGINT/SIGTERM.
@@ -657,4 +660,4 @@ Exit codes are **0** on accepted results, **2** for schema configuration errors,
657660
timeouts keep **1**; SIGINT and SIGTERM keep **130** and **143**.
658661
`StructuredOutputError.data.retries` records the number of corrective
659662
attempts used (initial attempt excluded), including corrective turns after a
660-
prose-only finish or a permission rejection.
663+
prose-only finish or a rejected tool call.

‎packages/cli/src/session/processor.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ export namespace SessionProcessor {
3737
}) {
3838
const toolcalls: Record<string, MessageV2.ToolPart> = {}
3939
let snapshot: string | undefined
40-
// Tools whose rejected permission stopped the loop.
40+
// Tools whose rejected permission or dismissed question stopped the loop.
4141
const blocked = new Set<string>()
4242
let attempt = 0
4343
let needsCompaction = false

‎packages/cli/src/session/prompt.ts‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ IMPORTANT:
6161
const STRUCTURED_OUTPUT_MISSING =
6262
"write the final result as JSON to a file in the working directory, then call StructuredOutput with its path"
6363

64-
const STRUCTURED_OUTPUT_DENIED = `a tool call was denied; ${STRUCTURED_OUTPUT_MISSING}`
64+
const STRUCTURED_OUTPUT_REJECTED = `a tool call was rejected; ${STRUCTURED_OUTPUT_MISSING}`
6565

6666
const STRUCTURED_OUTPUT_SYSTEM_PROMPT = `IMPORTANT: The user has requested structured output. Write your final result as JSON to a file in the working directory, then call the StructuredOutput tool with that file's path. Do NOT respond with plain text.`
6767

@@ -89,14 +89,15 @@ export namespace SessionPrompt {
8989
return !hasToolCalls(await load())
9090
}
9191

92-
// A schema turn that stops without a result costs one attempt and earns a
93-
// corrective reminder: a prose-only finish, or a headless permission
94-
// rejection (#135). The reminder names the denied tools, never their input.
92+
// A schema turn that stops without a result earns a corrective reminder: a
93+
// prose-only finish, or a rejected tool call (a permission auto-rejected in
94+
// headless mode, or a dismissed question; #135). The reminder names the
95+
// rejected tools, never their input.
9596
async function missingResult(processor: SessionProcessor.Info) {
9697
if (processor.blocked.length && !processor.message.error)
9798
return {
98-
diagnostic: { path: "", keyword: "denied", message: STRUCTURED_OUTPUT_DENIED },
99-
reminder: `Permission denied for ${processor.blocked.join(", ")}. Using what you already have, ${STRUCTURED_OUTPUT_MISSING}`,
99+
diagnostic: { path: "", keyword: "rejected", message: STRUCTURED_OUTPUT_REJECTED },
100+
reminder: `Tool call rejected: ${processor.blocked.join(", ")}. Using what you already have, ${STRUCTURED_OUTPUT_MISSING}`,
100101
}
101102
if (await missingStructuredOutput(processor.message, () => MessageV2.parts(processor.message.id)))
102103
return {
@@ -818,6 +819,7 @@ export namespace SessionPrompt {
818819
system.push(STRUCTURED_OUTPUT_SYSTEM_PROMPT)
819820
}
820821

822+
const prior = attempts
821823
const result = await processor.process({
822824
user: lastUser,
823825
agent,
@@ -892,18 +894,20 @@ export namespace SessionPrompt {
892894

893895
const missing = contract ? await missingResult(processor) : undefined
894896
reminder = missing?.reminder
895-
if (missing) await rejection([missing.diagnostic])
897+
// A rejected StructuredOutput call in the same step already spent its attempt.
898+
const missed = missing !== undefined && attempts === prior
899+
if (missed) await rejection([missing.diagnostic])
896900

897901
if (contract && !processor.message.error && rejected > contract.format.retryCount) {
898902
processor.message.error = new MessageV2.StructuredOutputError({
899-
message: reminder
903+
message: missed
900904
? "Model did not produce structured output"
901905
: "Structured output corrective attempts exhausted",
902906
retries: Math.max(0, attempts - 1),
903907
}).toObject()
904908
await Session.updateMessage(processor.message)
905909
await Bus.publish(Session.Event.Error, { sessionID, error: processor.message.error })
906-
outcome = { status: "failed", reason: reminder ? "missing" : "exhausted", attempts }
910+
outcome = { status: "failed", reason: missed ? "missing" : "exhausted", attempts }
907911
break
908912
}
909913

‎packages/cli/test/cli/run-output-schema.test.ts‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -763,8 +763,8 @@ test("JSON mode without output-result carries the accepted value only in termina
763763
])
764764
}, 25000)
765765

766-
const DENIED_REMINDER =
767-
"<system-reminder>Permission denied for read. Using what you already have, write the final result as JSON to a file in the working directory, then call StructuredOutput with its path</system-reminder>"
766+
const REJECTED_REMINDER =
767+
"<system-reminder>Tool call rejected: read. Using what you already have, write the final result as JSON to a file in the working directory, then call StructuredOutput with its path</system-reminder>"
768768

769769
test("headless permission rejection gets a corrective turn that can still succeed", async () => {
770770
const result = await run({
@@ -778,8 +778,8 @@ test("headless permission rejection gets a corrective turn that can still succee
778778
{ type: "function", function: { name: "StructuredOutput" } },
779779
])
780780
const messages = result.requests[1].messages as { role: string; content: unknown }[]
781-
expect(messages.filter((message) => JSON.stringify(message.content).includes("Permission denied"))).toEqual([
782-
{ role: "user", content: DENIED_REMINDER },
781+
expect(messages.filter((message) => JSON.stringify(message.content).includes("Tool call rejected"))).toEqual([
782+
{ role: "user", content: REJECTED_REMINDER },
783783
])
784784
expect(result.result).toBe(JSON.stringify({ result: "accepted" }, null, 2) + "\n")
785785
expect(result.events.filter((event) => event.type === "permission_rejected")).toHaveLength(1)
@@ -791,9 +791,9 @@ test("headless permission rejection gets a corrective turn that can still succee
791791
errors: [
792792
{
793793
path: "",
794-
keyword: "denied",
794+
keyword: "rejected",
795795
message:
796-
"a tool call was denied; write the final result as JSON to a file in the working directory, then call StructuredOutput with its path",
796+
"a tool call was rejected; write the final result as JSON to a file in the working directory, then call StructuredOutput with its path",
797797
},
798798
],
799799
},
@@ -818,9 +818,9 @@ test("repeated headless permission rejections spend the budget as missing output
818818
.filter((event) => event.type === "structured_output_rejected")
819819
.map((event) => [event.attempt, (event.errors as { keyword: string }[])[0].keyword]),
820820
).toEqual([
821-
[1, "denied"],
822-
[2, "denied"],
823-
[3, "denied"],
821+
[1, "rejected"],
822+
[2, "rejected"],
823+
[3, "rejected"],
824824
])
825825
expect(result.events.filter((event) => event.type === "structured_output")).toMatchObject([
826826
{ status: "failed", reason: "missing", attempts: 3 },

‎packages/cli/test/session/output-schema.test.ts‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ async function run(
3333
queued?: boolean
3434
followup?: boolean
3535
denied?: boolean
36+
beside?: boolean
3637
invalid?: boolean
3738
} = {},
3839
) {
@@ -92,10 +93,14 @@ async function run(
9293
: { name: "StructuredOutput", arguments: pathArgs(directory, args, `so_${requests.length}_${index}.json`) },
9394
}))
9495
if ((options.queued || options.denied) && requests.length === 1) {
95-
calls[0].function = {
96+
const read = {
9697
name: "read",
9798
arguments: JSON.stringify({ filePath: options.denied ? "/outside-fixture.txt" : "aictrl.json" }),
9899
}
100+
// `beside` keeps the first StructuredOutput call and adds the read next to it.
101+
if (options.beside)
102+
calls.push({ index: calls.length, id: `read_${requests.length}`, type: "function", function: read })
103+
else calls[0].function = read
99104
}
100105
const chunks = [
101106
{ choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }] },
@@ -620,7 +625,7 @@ test("permission rejection with zero retries spends the only attempt as missing
620625
data: { message: "Model did not produce structured output", retries: 0 },
621626
})
622627
expect(result.info.structured).toBeUndefined()
623-
expect(result.rejected).toMatchObject([{ attempt: 1, maxAttempts: 1, errors: [{ path: "", keyword: "denied" }] }])
628+
expect(result.rejected).toMatchObject([{ attempt: 1, maxAttempts: 1, errors: [{ path: "", keyword: "rejected" }] }])
624629
expect(result.outcomes).toEqual([{ status: "failed", reason: "missing", attempts: 1 }])
625630
}, 15000)
626631

@@ -631,15 +636,30 @@ test("permission rejection gets a named corrective turn before an accepted resul
631636
"required",
632637
{ type: "function", function: { name: "StructuredOutput" } },
633638
])
634-
// The reminder names the denied tool but never the arguments the model sent.
639+
// The reminder names the rejected tool but never the arguments the model sent.
635640
const reminder = JSON.stringify((result.requests[1].messages as unknown[]).at(-1))
636-
expect(reminder).toContain("Permission denied for read. Using what you already have")
641+
expect(reminder).toContain("Tool call rejected: read. Using what you already have")
637642
expect(reminder).not.toContain("outside-fixture")
638643
expect(result.info.structured).toEqual({ result: "accepted" })
639-
expect(result.rejected).toMatchObject([{ attempt: 1, maxAttempts: 3, errors: [{ keyword: "denied" }] }])
644+
expect(result.rejected).toMatchObject([{ attempt: 1, maxAttempts: 3, errors: [{ keyword: "rejected" }] }])
640645
expect(result.outcomes).toEqual([{ status: "accepted", attempts: 2, value: { result: "accepted" } }])
641646
}, 15000)
642647

648+
test("permission rejection beside an invalid StructuredOutput call spends one attempt", async () => {
649+
const result = await run(["{}", '{"result":"accepted"}'], 1, { denied: true, beside: true })
650+
expect(result.requests).toHaveLength(2)
651+
expect(result.requests[1].tool_choice).toEqual({ type: "function", function: { name: "StructuredOutput" } })
652+
expect(result.rejected).toMatchObject([{ attempt: 1, maxAttempts: 2, errors: [{ keyword: "required" }] }])
653+
expect(result.outcomes).toEqual([{ status: "accepted", attempts: 2, value: { result: "accepted" } }])
654+
}, 15000)
655+
656+
test("permission rejection beside an invalid call with no budget left reports exhaustion", async () => {
657+
const result = await run(["{}"], 0, { denied: true, beside: true })
658+
expect(result.requests).toHaveLength(1)
659+
expect(result.rejected).toMatchObject([{ attempt: 1, errors: [{ keyword: "required" }] }])
660+
expect(result.outcomes).toEqual([{ status: "failed", reason: "exhausted", attempts: 1 }])
661+
}, 15000)
662+
643663
test("permission rejection without a schema still stops the loop", async () => {
644664
const result = await run(["prose"], 0, { denied: true, plain: true })
645665
expect(result.requests).toHaveLength(1)

0 commit comments

Comments
 (0)