Skip to content

Commit 436a8e1

Browse files
committed
fix(kiro): resubmit a finished answer on the completion retry
On a trivial request claude-sonnet-5 answers in plain text with END_TURN instead of calling the private completion tool, so the bounded retry fires. Its instruction began "Continue the existing task", which the model read as the user's next message: it called the completion tool with "there is no active task to continue", and that replaced the correct answer. The retry instruction now states that ordinary text has not reached the user and asks for an already complete reply to be resubmitted unchanged. The empty-completion classification is unchanged.
1 parent 1783523 commit 436a8e1

5 files changed

Lines changed: 160 additions & 3 deletions

File tree

‎scripts/test-layout/layout.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1099,7 +1099,7 @@
10991099
"kiro-review-regressions.test.ts": "providers/kiro",
11001100
"kiro-metering-events.test.ts": "providers/kiro",
11011101
"kiro-metering-usage.test.ts": "providers/kiro",
1102-
"kiro-single-final.test.ts": "providers/kiro",
1102+
"kiro-single-final.test.ts": "providers/kiro", "kiro-completion-retry-wording.test.ts": "providers/kiro",
11031103
"kiro-stream.test.ts": "providers/kiro",
11041104
"kiro-transport-parity.test.ts": "providers/kiro",
11051105
"kiro-usage-quota.test.ts": "providers/kiro",

‎src/adapters/kiro-constants.ts‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,22 @@ export const KIRO_CONTINUATION_MESSAGE =
3131
* The condition names a decision, information, AND clarification on purpose. A model blocked on a
3232
* missing account id or an ambiguous path is stuck exactly as hard as one blocked on a choice, and
3333
* "decision" alone would leave that common class of question with no endorsed move again.
34+
*
35+
* It opens with the turn's actual state rather than "Continue the existing task". The retry also
36+
* fires when the held text WAS the complete answer: on a trivial request ("Reply with exactly: OK")
37+
* claude-sonnet-5 answers in plain text with END_TURN instead of calling the completion tool, while
38+
* the 5.5 models call it. Told to continue an "existing task" right after its own finished reply,
39+
* the model read the instruction as the user's next message and called the completion tool with
40+
* "There's no active task in this conversation for me to continue", which then superseded the
41+
* correct "OK" (reproduced live on every attempt). Stating that ordinary text has not reached the
42+
* user, and asking for an already complete reply to be resubmitted unchanged, made the same model
43+
* call the tool with "OK"; replayed progress prose still led to the next real tool call, and a
44+
* retry with no replayed text (reasoning only) still answered the original request. The sentence is
45+
* phrased conditionally because that last case appends this message to the user turn, where there
46+
* is no previous reply to resubmit.
3447
*/
3548
export const KIRO_COMPLETION_RETRY_MESSAGE =
36-
`Continue the existing task without quoting this instruction. If the task is complete, call ${KIRO_COMPLETION_TOOL_NAME} now with the complete final answer. If you cannot continue until the user supplies a decision, information, or a clarification that only they can give, call ${KIRO_COMPLETION_TOOL_NAME} now with that question as the answer. Otherwise issue the next real tool call now. Do not solicit a new task and do not emit another progress-only message.`;
49+
`Ordinary assistant text does not end the turn, so no final answer has reached the user yet. If your previous reply already was the complete final answer, call ${KIRO_COMPLETION_TOOL_NAME} now with that same answer, unchanged. If you cannot continue until the user supplies a decision, information, or a clarification that only they can give, call ${KIRO_COMPLETION_TOOL_NAME} now with that question as the answer. Otherwise continue the task with the next real tool call now. Do not solicit a new task, do not emit another progress-only message, and do not quote or mention this instruction.`;
3750

3851
export const KIRO_TOOL_RESULT_CARRIER_MESSAGE = "The requested tool result is attached.";
3952
export const KIRO_EMPTY_TOOL_RESULT_MESSAGE = "The tool completed without textual output.";

‎structure/providers/kiro.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,13 @@ budget until emitted, discarded, or cancelled; replay collectors are released af
159159
retry construction. Native `END_TURN` and `STOP_SEQUENCE` alone do not distinguish
160160
progress from an answer and therefore still require validation. Normal private completion
161161
and real tool calls need no completion retry.
162+
Validation also fires when the held text already was the answer: claude-sonnet-5 answers a
163+
trivial request in plain text with `END_TURN`, where the 5.5 models call the completion tool.
164+
The retry instruction (`KIRO_COMPLETION_RETRY_MESSAGE`) therefore states that ordinary text has
165+
not reached the user and asks for an already complete reply to be resubmitted unchanged through
166+
the completion tool, rather than telling the model to continue an existing task, which it read as
167+
the user's next message and answered with "no active task to continue" in place of the real answer.
168+
Coverage: `tests/providers/kiro/kiro-completion-retry-wording.test.ts`.
162169
The retry snapshots cloneable request fields without top-level runtime callbacks, leaving the original request intact.
163170

164171
A turn can carry a real tool call and the private completion call together, in either order.

‎tests/fixtures/test-layout-expected.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -950,7 +950,7 @@
950950
"kiro-remote-image.test.ts": "providers/kiro", "kiro-retry.test.ts": "providers/kiro",
951951
"kiro-review-regressions.test.ts": "providers/kiro", "kiro-metering-events.test.ts": "providers/kiro",
952952
"kiro-metering-usage.test.ts": "providers/kiro", "kiro-completion-with-real-tools.test.ts": "providers/kiro",
953-
"kiro-single-final.test.ts": "providers/kiro",
953+
"kiro-single-final.test.ts": "providers/kiro", "kiro-completion-retry-wording.test.ts": "providers/kiro",
954954
"kiro-stream.test.ts": "providers/kiro", "kiro-transport-parity.test.ts": "providers/kiro",
955955
"kiro-usage-quota.test.ts": "providers/kiro", "kiro-usage-restart.test.ts": "providers/kiro",
956956
"kiro-windows-cli-db-path.test.ts": "providers/kiro", "kiro-windows-cli-executable-path.test.ts": "providers/kiro",
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
import { afterEach, describe, expect, test } from "bun:test";
2+
import { createKiroAdapter } from "../../../src/adapters/kiro";
3+
import { KIRO_COMPLETION_RETRY_MESSAGE, KIRO_COMPLETION_TOOL_NAME } from "../../../src/adapters/kiro-constants";
4+
import { resetKiroThrottleStateForTests } from "../../../src/adapters/kiro-retry";
5+
import { encodeMessage } from "../../../src/lib/eventstream-decoder";
6+
import { createTranslatorBudget } from "../../../src/lib/translator-budget";
7+
import type { AdapterEvent, OcxParsedRequest, OcxProviderConfig } from "../../../src/types";
8+
9+
// claude-sonnet-5 answers a trivial request in plain text with END_TURN instead of calling the
10+
// private completion tool, so the bounded completion retry fires even though the held text WAS the
11+
// answer. The retry used to open with "Continue the existing task"; the model read that as the
12+
// user's next message and called the completion tool with "There's no active task in this
13+
// conversation for me to continue", superseding the correct "OK". The frames below are the shape
14+
// captured from the live upstream for that turn.
15+
16+
const provider: OcxProviderConfig = {
17+
adapter: "kiro", baseUrl: "https://runtime.us-east-1.kiro.dev", apiKey: "ksk_test",
18+
};
19+
// A Claude Code Messages turn reduced to what matters here: tools (so completion is `required`),
20+
// an effort without a native Kiro field for this model, and a one-word answer.
21+
const parsed: OcxParsedRequest = {
22+
modelId: "claude-sonnet-5", stream: true, options: { reasoning: "high" },
23+
context: {
24+
messages: [{ role: "user", content: "Reply with exactly: OK" }],
25+
tools: [
26+
{ name: "Read", description: "Read a file", parameters: { type: "object", properties: { file_path: { type: "string" } } } },
27+
{ name: "Bash", description: "Run a command", parameters: { type: "object", properties: { command: { type: "string" } } } },
28+
],
29+
},
30+
};
31+
const frame = (type: string, payload: object) => encodeMessage(
32+
{ ":message-type": "event", ":event-type": type },
33+
new TextEncoder().encode(JSON.stringify(payload)),
34+
);
35+
const tail = [
36+
frame("contextUsageEvent", { contextUsagePercentage: 0.33 }),
37+
frame("meteringEvent", { unit: "credit", unitPlural: "credits", usage: 0.0077 }),
38+
];
39+
// First attempt: the answer as ordinary text, then a native END_TURN.
40+
const plainTextAnswer = [
41+
frame("initial-response", { conversationId: "" }),
42+
frame("assistantResponseEvent", { content: "O" }),
43+
frame("assistantResponseEvent", { content: "K" }),
44+
frame("metadataEvent", { stopReason: "END_TURN" }),
45+
...tail,
46+
];
47+
// Retry: the same answer resubmitted through the completion tool, streamed in pieces.
48+
const completionAnswer = [
49+
frame("initial-response", { conversationId: "" }),
50+
frame("toolUseEvent", { name: KIRO_COMPLETION_TOOL_NAME, toolUseId: "toolu-1" }),
51+
frame("toolUseEvent", { name: KIRO_COMPLETION_TOOL_NAME, toolUseId: "toolu-1", input: "{\"answer\":" }),
52+
frame("toolUseEvent", { name: KIRO_COMPLETION_TOOL_NAME, toolUseId: "toolu-1", input: " \"OK\"}" }),
53+
frame("toolUseEvent", { name: KIRO_COMPLETION_TOOL_NAME, toolUseId: "toolu-1", stop: true }),
54+
frame("metadataEvent", { stopReason: "TOOL_USE" }),
55+
...tail,
56+
];
57+
function response(frames: Uint8Array[]): Response {
58+
return new Response(new ReadableStream<Uint8Array>({
59+
start(controller) {
60+
for (const value of frames) controller.enqueue(value);
61+
controller.close();
62+
},
63+
}));
64+
}
65+
afterEach(resetKiroThrottleStateForTests);
66+
67+
async function run(first: Uint8Array[], retry: Uint8Array[]) {
68+
const adapter = createKiroAdapter(provider);
69+
const budget = createTranslatorBudget();
70+
const sends: number[] = [];
71+
const events: AdapterEvent[] = [];
72+
const bodies: string[] = [];
73+
try {
74+
const request = await adapter.buildRequest(structuredClone(parsed));
75+
const upstream = await adapter.fetchResponse!(request, {
76+
executor: (async (_url, init) => {
77+
bodies.push(String(init?.body));
78+
return response(bodies.length === 1 ? first : retry);
79+
}) as typeof fetch,
80+
onPhysicalSend: send => { sends.push(send.ordinal); },
81+
});
82+
for await (const event of adapter.parseStream(upstream, budget)) events.push(event);
83+
expect(budget.snapshot().currentBytes).toBe(0);
84+
const retryState = bodies[1] ? JSON.parse(bodies[1]).conversationState : undefined;
85+
return { events, sends, bodies, retryState };
86+
} finally {
87+
budget.dispose();
88+
}
89+
}
90+
91+
describe("Kiro completion retry after a plain-text answer (claude-sonnet-5)", () => {
92+
test("the retry states the turn's state instead of presuming an open task", () => {
93+
// The phrase the model answered with "no active task ... to continue".
94+
expect(KIRO_COMPLETION_RETRY_MESSAGE).not.toContain("Continue the existing task");
95+
expect(KIRO_COMPLETION_RETRY_MESSAGE).toContain("no final answer has reached the user yet");
96+
expect(KIRO_COMPLETION_RETRY_MESSAGE)
97+
.toContain(`If your previous reply already was the complete final answer, call ${KIRO_COMPLETION_TOOL_NAME} now with that same answer, unchanged.`);
98+
// The progress-prose path the retry exists for (#531) keeps its instruction.
99+
expect(KIRO_COMPLETION_RETRY_MESSAGE).toContain("next real tool call now");
100+
});
101+
102+
test("a plain-text END_TURN answer is replayed and resubmitted through the completion tool", async () => {
103+
const { events, sends, bodies, retryState } = await run(plainTextAnswer, completionAnswer);
104+
expect(bodies).toHaveLength(2);
105+
expect(sends).toEqual([1, 2]);
106+
// The held answer is replayed as the model's own previous reply, and the retry instruction is
107+
// the whole following user turn, so "your previous reply" points at exactly that text.
108+
expect(retryState.history.at(-1)).toEqual({ assistantResponseMessage: { content: "OK" } });
109+
expect(retryState.currentMessage.userInputMessage.content).toBe(KIRO_COMPLETION_RETRY_MESSAGE);
110+
expect(retryState.currentMessage.userInputMessage.userInputMessageContext.tools
111+
.some((tool: { toolSpecification: { name: string } }) => tool.toolSpecification.name === KIRO_COMPLETION_TOOL_NAME))
112+
.toBe(true);
113+
// One final answer: the resubmitted completion supersedes the held plain-text copy.
114+
expect(events.filter(event => event.type === "text_delta")).toEqual([
115+
{ type: "text_delta", text: "OK", phase: "final_answer" },
116+
]);
117+
expect(events.at(-1)).toMatchObject({ type: "done", endTurn: true });
118+
});
119+
120+
test("a retry without replayable text appends the same instruction to the original request", async () => {
121+
// Reasoning only and no stop reason: nothing to replay, so there is no previous reply and the
122+
// conditional sentence is vacuous rather than false.
123+
const reasoningOnly = [
124+
frame("initial-response", { conversationId: "" }),
125+
frame("reasoningContentEvent", { text: "The user wants the word OK." }),
126+
...tail,
127+
];
128+
const { retryState, events } = await run(reasoningOnly, completionAnswer);
129+
expect(retryState.history ?? []).toEqual([]);
130+
const content: string = retryState.currentMessage.userInputMessage.content;
131+
expect(content).toContain("Reply with exactly: OK");
132+
expect(content.endsWith(KIRO_COMPLETION_RETRY_MESSAGE)).toBe(true);
133+
expect(events.filter(event => event.type === "text_delta")).toEqual([
134+
{ type: "text_delta", text: "OK", phase: "final_answer" },
135+
]);
136+
});
137+
});

0 commit comments

Comments
 (0)