Skip to content

Commit afc1cc5

Browse files
authored
fix(kiro): let a blocking question be the final answer (#3031)
* docs(devlog): record why the Kiro terminal-wording fix failed and what replaces it Round one added terminal semantics to both injected surfaces and the defect recurred anyway: the serving proxy (PID 55727, v2.37.0, started 09:23:37) ran from a checkout with that commit as an ancestor, and the failure came back at 09:27. The mechanism was never a second turn continuing past its answer. A 1454-char answer-shaped message and the next exec call left one inference 4ms apart with sendCount 1. The text proves it: a permission question glued with no separator to a sentence that overrides it. The model asked and answered itself. Root cause is expressibility, not enforcement. The contract described two states (still working, fully done) for a model that has three, and the retry message read as a ban on asking. Continuing to work was the only endorsed move left. Records the measured regression boundary that rules out every gating design: across 644 rollouts, same-inference prose plus a real tool call occurs 26 times, 4 question-tailed at 1329-1938 chars and 22 legitimate at 608-3141, overlapping completely. * fix(kiro): let a blocking question be the final answer so the model stops answering itself Round one made the completion tool terminal and the defect came back anyway. Terminality was never the gap. The failing message and the next exec call left ONE inference 4ms apart with sendCount 1. The text shows two emissions merged: a permission question glued with no separator to a sentence that overrides it. The model asked the user something and then answered itself, in one breath. The contract described two states -- still working, fully done -- for a model that has three. Prose does not end the turn, the completion tool meant the task was finished, and a real tool call continues the turn, so "I need a decision before I can go on" was not expressible. The retry message closed the last door: "Do not ask the user for another task" was meant to stop soliciting new work, but it reads as a ban on asking anything, and it is the one instruction the model sees at the moment it fails to complete. Continuing to work was the only endorsed move left. So the third state is now expressible on every surface that describes when to complete: the injected contract, the retry message, the completion tool's schema description, and its answer property. The schema half mattered -- it is what the model reads while CHOOSING, and it admitted only "fully complete", which round one left untouched. No adapter gate, on evidence. Across 644 rollouts the same-inference prose-plus-tool shape occurs 26 times: 4 question-tailed at 1329-1938 chars, 22 legitimate at 608-3141. They overlap completely, and at flushOpen the adapter knows only that a non-completion tool was emitted, its identity, and its arguments. Every gate would be a coin flip on whether a user sees their agent's work. Ask-tool isolation was measured unreachable (8 ask calls, ask-then-another-tool 0 times) and the predecessor already shipped and reverted one guard for an unreachable path. Tests: the contract assertions were driven red against the old strings first. The non-regression test is parameterized over the measured overlap and passes byte-identically with the constants stashed and restored. Two hostile-catalog tests pin the contract arriving complete at both budget extremes, each asserting its own precondition so neither can pass while charging less than it claims. This is influence, not enforcement. Kiro accepts only automatic or no tool choice, so no typed pause protocol can be forced upstream, and the good and bad event streams are observationally identical at the adapter. The tests prove the contract is delivered, not that the model obeys it. * fix(kiro): cover information and clarification, not just decisions Review findings from PR #3031. The pause clause said "blocked on a decision only the user can make". A model that cannot continue without an account id, a missing path, or a disambiguation is stuck exactly as hard, but that is not a decision, so the clause did not reach it and the original ask-and-self-answer path stayed open for a common class of question. The trigger now names a decision, information, or a clarification on all four surfaces: the injected contract, the retry message, the completion tool's schema description, and its answer property. The non-regression test asserted counts and payloads but not positions, so a reordered stream or an early done followed by a second one would still have passed. It now pins commentary before the call starts, every argument delta inside the call, and exactly one terminal after it closes. docs-site gains the pause semantics in the adapter reference: a blocking question arrives as final text with the turn ended, not as commentary and not as a client tool call. The seven translated locales get the same statement so none of them contradicts the English source.
1 parent 870a2ad commit afc1cc5

14 files changed

Lines changed: 496 additions & 6 deletions

File tree

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Kiro post-answer tool calls, round two: why the prompt fix failed
2+
3+
The user reported the same defect a second time, with a hard pair of
4+
requirements: never recur, and do not regress anything.
5+
6+
## The first fix is live and did not work
7+
8+
PR #3012 (`f5a625cf3`) appended terminal semantics to both injected surfaces.
9+
That code was serving when the defect recurred:
10+
11+
| Fact | Value |
12+
| --- | --- |
13+
| Serving proxy | PID 55727, port 10100, version 2.37.0 |
14+
| Proxy start | 09:23:37 |
15+
| Adapter file mtime | 01:28 (same day, before the start) |
16+
| `f5a625cf3` ancestry | ancestor of the serving HEAD |
17+
| Failure | 09:27 |
18+
19+
So prompt wording is not a sufficient mechanism for this defect. That is the
20+
finding that governs this unit.
21+
22+
## What actually happened
23+
24+
A 1454-char answer-shaped message and the next `exec` call came out of ONE
25+
inference, 4ms apart (message `00:27:23.114`, call `00:27:23.118`), with
26+
`sendCount: 1` and no second upstream request. This was never "the model keeps
27+
working after the turn ended" — there was no second turn to keep working in.
28+
29+
The message text is the proof. It ends:
30+
31+
> ...이 호출 경계 동작은 문서에 아직 정확히 안 들어가 있습니다. 넣어둘까요.레퍼런스에
32+
> 있는 "scope persists" 서술이 실측과 어긋나니 그 부분부터 고치겠습니다.
33+
34+
A permission question, glued with no separator to a sentence that overrides the
35+
question and proceeds. Two emissions merged into one message: the model asked
36+
the user something and then answered itself, inside one inference.
37+
38+
The adapter behaved exactly as designed. `defer()` holds staged text, a real
39+
tool call proves the turn continues, and the staged prose is released as
40+
`commentary`.
41+
42+
## Root cause: there is no sanctioned way to pause
43+
44+
In `required` mode the model has three expressible moves and none of them is
45+
"I have a question, hold for the user":
46+
47+
- Ordinary prose does not end the turn, by explicit contract.
48+
- The completion tool means "the task is fully complete and no more work or tool
49+
calls are needed" — which a pending question is not.
50+
- A real tool call continues the turn.
51+
52+
`KIRO_COMPLETION_RETRY_MESSAGE` closes the last door explicitly: *"Do not ask
53+
the user for another task or emit another progress-only message."*
54+
55+
So a model that wants to ask has no explicitly endorsed blocking-question move.
56+
Pausing is mechanically expressible — the ask tool is advertised and works — but
57+
nothing in the injected contract endorses it, and the one instruction the model
58+
sees at the moment it fails to complete reads as a ban on asking. Continuing to
59+
work is the only move the contract actively endorses, which is what it did.
60+
61+
## The regression boundary, measured
62+
63+
Across 644 Kiro rollouts, same-inference prose (>=600 chars) followed by a real
64+
tool call occurs 26 times. Only 4 have a tail that asks the user something. The
65+
other 22 are ordinary progress narration and must keep working unchanged.
66+
67+
Prose length cannot separate them:
68+
69+
- 4 question-tailed: 1329, 1454, 1697, 1938 chars.
70+
- 22 legitimate: 608 ... 3141 chars.
71+
72+
The ranges overlap completely. Any length or volume threshold that catches the
73+
four also catches legitimate narration, and any threshold that spares the 22
74+
also spares the defect. There is no structural discriminator for the general
75+
prose-plus-tool shape, which an independent read-only design audit confirmed
76+
from the code: at `flushOpen` the adapter knows only that a non-completion tool
77+
was emitted, its restored identity, and its arguments. `stopReason` cannot help
78+
either — Kiro emits `END_TURN` for progress prose, and the adapter deliberately
79+
ignores it when a real tool exists.
80+
81+
## Rejected candidates
82+
83+
**Gate the answer-shaped prose.** Rejected: it requires a text heuristic to tell
84+
the 4 from the 22, and the measurement above shows no non-heuristic signal
85+
exists. A regex would decide whether the user sees their agent's work.
86+
87+
**Discard prose when a real tool follows** (what `consumeSupersededByCompletion`
88+
does on the completion path). Rejected: on the real-tool path no replacement
89+
answer ever arrives, so all 22 legitimate progress messages would silently
90+
vanish rather than arrive later.
91+
92+
**Isolate the ask tool** (relay `request_user_input`, fail closed on any later
93+
real tool from the same inference). Rejected as the primary mechanism: measured
94+
8 `request_user_input` calls across 644 rollouts, and the
95+
ask-then-another-tool shape occurs **0** times. It guards a case that does not
96+
happen. Shipping a guard for an unreachable path with a test that cannot fail is
97+
the exact trap this unit's predecessor already fell into once.
98+
99+
## What the evidence does support
100+
101+
`request_user_input` already terminates a Kiro turn cleanly. Measured three
102+
times in the failing session itself: staged prose, then the ask call, then the
103+
turn yields and the human answers. The pause path exists and works — the model
104+
simply did not take it, because the contract never told it to and the retry
105+
message reads as a ban on asking.
106+
107+
The completion channel is also content-agnostic: a completion answer yields
108+
`final_answer` and `done(endTurn: true)` regardless of what the answer says. A
109+
question delivered there ends the turn correctly today.
110+
111+
So the gap is not enforcement, it is expressibility. The contract describes two
112+
states (still working, fully done) for a model that has three (still working,
113+
done, blocked on the user). The fix is to make the third state expressible and
114+
to stop forbidding it.
115+
116+
## Honest statement of limits
117+
118+
This unit cannot make the defect impossible without model compliance, and it
119+
will not claim otherwise. Deterministic prevention would need a structural
120+
upstream signal that does not exist: Kiro accepts only automatic or no tool
121+
choice, so the proxy cannot force a typed progress/pause/complete protocol. The
122+
bad and good event streams are observationally equivalent at the adapter.
123+
124+
What this unit can do is remove the contradiction that made the defect the
125+
model's *only* endorsed move: make the blocked-on-user state expressible, and
126+
stop forbidding it. That is one change and it is testable. It introduces no
127+
adapter-side regression for the 22 — the non-regression test guards their
128+
structural output byte for byte — but it is a prompt change, so it can still
129+
shift what the model chooses to emit. Adapter behavior is pinned; model
130+
behavior is influenced, not fixed.
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# wp2 — make the blocked-on-user state expressible
2+
3+
One change, in the injected contract, carried on every surface that describes
4+
when to complete. It removes the contradiction that made continuing to work the
5+
model's only endorsed move. No adapter enforcement is added, because no
6+
non-heuristic signal exists to enforce on.
7+
8+
## A blocking question is a valid final answer
9+
10+
The pause state rides the channel that already terminates correctly. A
11+
completion answer yields `final_answer` and `done(endTurn: true)`
12+
**regardless of what the answer says** — verified in `kiro.ts` at the
13+
completion terminal. So a question delivered there ends the turn today, with no
14+
new event type, no new phase, and no change to `endTurn` semantics for real
15+
tools.
16+
17+
`KIRO_COMPLETION_INSTRUCTIONS` therefore gains one clause: when the model needs
18+
a decision from the user before it can continue, that question IS its final
19+
answer — deliver it through the completion tool and stop, instead of writing the
20+
question as prose and then answering itself.
21+
22+
The completion tool's own schema carries the same clause. That description is the
23+
surface the model reads while CHOOSING a tool, and it admitted only "fully
24+
complete"; a model holding a blocking question reads the tool as unavailable and
25+
keeps working, which is the defect. Round one changed only the terminality half of
26+
this description and left the eligibility half narrow, which is part of why prose
27+
alone did not move the outcome. The `answer` property description is widened the
28+
same way, since it is what tells the model what may go in the field.
29+
30+
This is deliberately NOT conditioned on an ask tool being present. An earlier
31+
draft keyed it on `request_user_input` appearing in the emitted catalog, which
32+
would have named a tool that is absent from most turns and contradicted the
33+
nudge's rule that instruction-only names are not callable (the failure mode
34+
`0325a5afd` fixed for the catalog nudge). The completion tool is always present
35+
when this instruction is emitted — that is the condition under which the
36+
instruction exists at all — so the clause can never name something uncallable.
37+
38+
`KIRO_COMPLETION_RETRY_MESSAGE` currently ends with *"Do not ask the user for
39+
another task or emit another progress-only message."* In context that forbids
40+
soliciting NEW work, but it reads as a blanket ban on asking, and it is the one
41+
instruction the model sees at exactly the moment it failed to complete. It is
42+
narrowed to keep forbidding a request for another task while making a blocking
43+
question an explicitly acceptable final answer.
44+
45+
Both sites are compared by constant, never by inline literal
46+
(`currentUim.content !== KIRO_COMPLETION_RETRY_MESSAGE`), and both pinning
47+
tests assert against the constant, so editing the text cannot desynchronize the
48+
equality checks that keep the bounded retry from stacking.
49+
50+
This depends on model compliance and is labeled as such. It is the reason this
51+
unit does not promise impossibility: the tests prove the contract is delivered,
52+
not that the model obeys it.
53+
54+
## Ask-tool isolation: considered and dropped
55+
56+
An earlier draft added a `flushOpen` invariant: once a turn emitted the ask
57+
tool, a later real tool call from the same inference would fail closed. It is not
58+
being implemented, for three independent reasons.
59+
60+
It is unreachable. Measured across 644 Kiro rollouts: 8 `request_user_input`
61+
calls, and the ask-then-another-real-tool shape occurs **0** times.
62+
63+
It cannot be keyed safely. `nameMap` aliases every REQUESTED tool before
64+
admission, and admission can omit tools, so `nameMap` membership does not prove
65+
a tool was emitted. Doing it correctly means threading an explicit emitted-ask
66+
identity through `buildRequest`, the fallback state, and `parseKiroStream`.
67+
68+
It can false-positive. `OcxTool` carries no "asks the user" marker, and the
69+
parser reduces ordinary function tools to the same generic shape, so an unrelated
70+
bare function named `request_user_input` is indistinguishable from Codex's ask
71+
tool. Claiming it "cannot affect any ordinary tool" was an overclaim.
72+
73+
Three costs and a measured zero benefit. The previous unit already shipped and
74+
then reverted a guard for an unreachable path; this one declines it before the
75+
commit instead of after.
76+
77+
## What is NOT changed, and why
78+
79+
No prose-shape gate. No length threshold. No discard of staged commentary. The
80+
measurement in `000_research.md` shows the 4 defect cases (1329-1938 chars) and
81+
the 22 legitimate ones (608-3141) overlap completely, so every such rule is a
82+
coin flip on whether the user sees their agent's work.
83+
84+
No change to `src/router.ts`, `src/server/lifecycle.ts`, or
85+
`src/server/responses/core.ts`: the Lab core boundary is unrelated and
86+
`tests/core-lab-boundary.test.ts` guards it. No change to the shared
87+
`tool-catalog-nudge` sentence or its pinned test.
88+
89+
## Tests
90+
91+
**Red-first, contract shape:** every emitted completion contract carries the
92+
pause clause, and the retry message permits a blocking question as a final
93+
answer. Driven red against the current strings before the edit.
94+
95+
**Non-regression, the 22:** staged progress prose followed by `exec` still
96+
yields the prose as `commentary` and `done(endTurn: false)` with the tool call
97+
relayed, byte-identical to today, parameterized across the measured lengths
98+
608 / 1329 / 1454 / 1938 / 3141. This is the test that protects the user's
99+
"회귀없도록" requirement, and it must pass unchanged both before and after the
100+
diff.
101+
102+
## Budget truncation: measured unreachable, pinned by test
103+
104+
The completion contract is charged last against
105+
`MAX_KIRO_INJECTED_INSTRUCTION_CHARS` (16384), so in principle a large enough
106+
injected context could slice the pause clause mid-sentence. Measured against a
107+
hostile catalog (80 requested tools, 64-character names, 6000-character
108+
descriptions):
109+
110+
| Charged item | Chars |
111+
| --- | --- |
112+
| Omission notice | 922 |
113+
| Catalog nudge | 2432 |
114+
| Total before the contract | 3354 |
115+
| Headroom | 13030 |
116+
| Contract | 696 |
117+
118+
Both charged inputs are structurally capped — the notice names at most 12 tools,
119+
the nudge is bounded by the 48-tool and 64-character limits — and the caller's
120+
system prompt is not charged to this budget at all. So no reachable input
121+
approaches the slice boundary.
122+
123+
A reservation guard is therefore dead code, and the previous unit already proved
124+
its test would be vacuous: it passed with the guard removed. Instead this unit
125+
pins the property that makes truncation unreachable, at BOTH hostile extremes,
126+
asserting the contract arrives COMPLETE including its closing pause clause:
127+
128+
- **Largest nudge:** `MAX_KIRO_TOOL_COUNT` admitted tools with unique 64-character
129+
names and one-character descriptions. Every admitted tool is named in the nudge,
130+
so this maximizes the charged nudge. Nothing is omitted, which the test asserts
131+
so it cannot pass while silently charging less than it claims.
132+
- **Notice plus nudge:** oversized descriptions on twice the tool limit, which
133+
forces admission to omit and charges the omission notice on top of the nudge.
134+
The test asserts the notice is actually present for the same reason.
135+
136+
The 6000-character-description shape alone would not have been enough: it reduces
137+
the admitted count and therefore shrinks the nudge, so it never probes the
138+
largest charged input. Both tests fail if a future change lets the notice or the
139+
nudge grow without bound. A reservation's test could not detect its own removal.
140+
141+
## Verification
142+
143+
`bun run typecheck`, the focused Kiro suites, and `bun run privacy:scan`. The
144+
full local suite is excluded by explicit user instruction; CI covers it.

‎docs-site/src/content/docs/fr/reference/adapters.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ Le texte de l’assistant Kiro ne comporte pas, à lui seul, de phase fiable ind
125125

126126
`END_TURN`, `STOP_SEQUENCE` ou l’absence de motif d’arrêt peuvent emprunter le chemin de compatibilité. Les autres motifs explicites ont déjà interrompu l’inférence en amont ; l’adaptateur les signale donc au lieu de consommer une nouvelle requête au modèle. Une limite de jetons de sortie est présentée comme une sortie incomplète que le client peut poursuivre, tandis qu’un épuisement de la fenêtre de contexte devient une erreur de longueur de contexte non renouvelable plutôt qu’une sortie tronquée. Les arrêts dus au filtrage et aux garde-fous sont présentés comme des sorties filtrées incomplètes. Un arrêt `TOOL_USE` sans appel d’outil réel est signalé comme une contradiction, et non comme une progression.
127127

128-
Lorsqu’un outil client ordinaire est disponible, opencodex ajoute un outil privé `codex_kiro_final_answer` à la requête en amont. Le texte de progression est diffusé comme commentaire et ne peut pas clore le tour. L’adaptateur consomme l’appel privé, émet sa réponse comme texte final et n’expose jamais cet outil privé à Codex ou Claude Code. Le motif d’arrêt n’arrivant qu’à la fin du flux, le texte de l’assistant est retenu pendant un tour avec outils jusqu’au début d’un véritable appel d’outil ou jusqu’à la fin du flux. Il est alors libéré comme commentaire, sauf si l’outil privé a fourni la réponse finale. Lorsque le service auxiliaire de recherche Web est actif, les commentaires libérés continuent d’être diffusés avant l’événement terminal ; seuls les événements nécessaires pour déterminer si le modèle a demandé une recherche synthétique restent en mémoire tampon.
128+
Lorsqu’un outil client ordinaire est disponible, opencodex ajoute un outil privé `codex_kiro_final_answer` à la requête en amont. Le texte de progression est diffusé comme commentaire et ne peut pas clore le tour. L’adaptateur consomme l’appel privé, émet sa réponse comme texte final et n’expose jamais cet outil privé à Codex ou Claude Code. Le motif d’arrêt n’arrivant qu’à la fin du flux, le texte de l’assistant est retenu pendant un tour avec outils jusqu’au début d’un véritable appel d’outil ou jusqu’à la fin du flux. Il est alors libéré comme commentaire, sauf si l’outil privé a fourni la réponse finale. Lorsque le service auxiliaire de recherche Web est actif, les commentaires libérés continuent d’être diffusés avant l’événement terminal ; seuls les événements nécessaires pour déterminer si le modèle a demandé une recherche synthétique restent en mémoire tampon. Lorsque le modèle ne peut pas continuer sans une décision, une information ou une clarification que seul l’utilisateur peut fournir, le contrat lui demande de transmettre cette question via l’outil d’achèvement et de s’arrêter ; un tel tour arrive lui aussi comme `final_answer` avec le tour clos, et non comme un commentaire.
129129

130130
Si Kiro s’arrête sans appeler l’outil d’achèvement, l’adaptateur effectue une continuation. Les nouvelles tentatives ne contenant que du raisonnement conservent le tour utilisateur/résultat d’outil valide d’origine au lieu de fabriquer un message assistant vide ; la progression visible est rejouée avec une instruction non vide appartenant à l’adaptateur. Avant le transport, la conversation générée est contrôlée afin de vérifier l’alternance des rôles, l’absence de tours structurels vides et la correspondance des identifiants d’utilisation et de résultat d’outil. Une sortie d’outil vide reçoit un texte de remplacement neutre et non vide. La nouvelle tentative ne peut pas se répéter récursivement : si elle est vide ou ne contient que du raisonnement, elle est renvoyée comme incomplète mais renouvelable ; un véritable appel d’outil client maintient le tour ouvert. Une réponse de l’outil d’achèvement est toujours émise comme `final_answer`, même si elle répète exactement un commentaire antérieur, car l’exactitude de la phase prime sur la déduplication esthétique. Les requêtes sans outil conservent le comportement normal d’achèvement textuel.
131131

‎docs-site/src/content/docs/ja/reference/adapters.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,7 @@ filtered incomplete になります。実際のツール呼び出しを伴わな
150150
ツール有効ターンでは非公開の `codex_kiro_final_answer` を追加します。再試行は空の assistant/user ターンを
151151
生成せず、元の user/tool-result を保持し、送信前にロール交互性、空の構造メッセージ、tool use/result の対応を検証します。
152152
完了ツールの回答は以前の commentary と同じでも `final_answer` として送出します。
153+
ユーザーしか出せない判断・情報・確認が得られず先に進めない場合も、その質問を完了ツールで送って停止するよう契約が指示します。これも commentary ではなくターンを終えた `final_answer` として届きます。
153154

154155
### Reasoning effort
155156

‎docs-site/src/content/docs/ko/reference/adapters.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ commentary로 유지하고 비공개 완료 툴을 한 번 검증합니다.
163163
툴이 있는 턴에는 비공개 `codex_kiro_final_answer`를 추가합니다. 완료 재시도는 빈 assistant/user 턴을 만들지
164164
않고 원래 user/tool-result를 보존하며, 전송 전에 역할 교대·빈 구조 메시지·tool use/result 짝을 검증합니다.
165165
완료 툴 답변은 이전 commentary와 같더라도 `final_answer`로 내보냅니다.
166+
사용자만 줄 수 있는 결정·정보·설명이 없어 더 진행할 수 없을 때도 그 질문을 완료 툴로 보내고 멈추도록 계약이 지시합니다. 이때도 commentary가 아니라 턴이 끝난 `final_answer`로 도착합니다.
166167

167168
### Reasoning effort
168169

0 commit comments

Comments
 (0)