|
| 1 | +# Kiro post-final-answer tool calls — measurement and root cause |
| 2 | + |
| 3 | +Reported symptom, twice: routed through Kiro, the agent keeps issuing tool calls |
| 4 | +after its final response has already been delivered. |
| 5 | + |
| 6 | +## Hosts measured |
| 7 | + |
| 8 | +| Host | Proxy | Version | Checkout | Kiro attempt rows | |
| 9 | +| --- | --- | --- | --- | --- | |
| 10 | +| local (this machine) | PID 99470, port 10100 | 2.36.0 | primary source checkout | 4080 | |
| 11 | +| `macmini-cf` | PID 96671, port 10100 | 2.35.0 | `~/opencodex` | 0 | |
| 12 | + |
| 13 | +`macmini-cf` carries no Kiro attempt diagnostics at all, so every behavioral |
| 14 | +row below comes from the local 2.36.0 proxy. The remote host is one release |
| 15 | +behind and is not the reporting surface. |
| 16 | + |
| 17 | +## What the attempt rows say |
| 18 | + |
| 19 | +`ocx:kiro:attempt_complete` over the local log, bucketed: |
| 20 | + |
| 21 | +| Count | mode | sawText | sawRealTool | completionCalls | stopReason | |
| 22 | +| --- | --- | --- | --- | --- | --- | |
| 23 | +| 2643 | required | true | true | 0 | TOOL_USE | |
| 24 | +| 1400 | required | false | true | 0 | TOOL_USE | |
| 25 | +| 23 | required | true | false | 1 | TOOL_USE | |
| 26 | +| 10 | disabled | true | false | 0 | END_TURN | |
| 27 | +| 2 | required | false | false | 1 | TOOL_USE | |
| 28 | +| 1 | required | true | false | 0 | END_TURN | |
| 29 | +| 1 | text_fallback | false | false | 1 | TOOL_USE | |
| 30 | + |
| 31 | +4069 of 4080 attempts ran in `required` mode and every one of them ended with |
| 32 | +upstream `stopReason: TOOL_USE`. Only 25 attempts ever called the private |
| 33 | +completion tool. The model overwhelmingly prefers another tool call to the |
| 34 | +completion channel. |
| 35 | + |
| 36 | +## What is NOT the cause |
| 37 | + |
| 38 | +Two candidate mechanisms were ruled out with evidence rather than reading. |
| 39 | + |
| 40 | +Replayed history is not the cause. 532 client rollouts under |
| 41 | +`~/.codex/sessions/2026/08/{29,30}` were scanned for a `final_answer` message |
| 42 | +followed by a tool call with no intervening user turn. What actually follows a |
| 43 | +recorded `final_answer`: END 478, user message 131, developer message 5, tool |
| 44 | +call 0. The client never replays a post-answer tool call. |
| 45 | + |
| 46 | +The delivered-answer local terminal is not broken. Two live probes against the |
| 47 | +running proxy replayed a closed turn — once with `phase: "final_answer"` |
| 48 | +echoed, once without it, matching real Codex traffic — and both returned |
| 49 | +`output: []` with `end_turn: true` and added zero upstream Kiro requests. |
| 50 | +The guard added in `b557a8140`/`68eaf45d8` works. |
| 51 | + |
| 52 | +It has simply never been needed: `~/.opencodex/usage.jsonl` holds 25042 Kiro |
| 53 | +rows with zero `localTerminalReason` and zero `locallyAnswered`. Real turns |
| 54 | +never arrive already closed, because the client ends the turn itself. So the |
| 55 | +defect lives inside a live turn, not across turns. |
| 56 | + |
| 57 | +## Rejected first hypothesis |
| 58 | + |
| 59 | +The first diagnosis was that the model calls the completion tool, waits for a |
| 60 | +tool result that never arrives, and then calls another tool. An independent |
| 61 | +read-only audit refuted it with the parser: `flushOpen` consumes a valid |
| 62 | +completion call and records `completionAnswer` without emitting any tool-call |
| 63 | +event, the stream end yields the answer as `final_answer` followed by |
| 64 | +`done(endTurn: true)`, and `parseKiroStream` returns without another request. |
| 65 | +A completion call therefore terminates locally inside one inference; there is |
| 66 | +no later inference in which the model could "keep going". Mixed |
| 67 | +completion-plus-real-tool output in one inference also fails closed before any |
| 68 | +answer is delivered. |
| 69 | + |
| 70 | +That refutation is correct, and it narrows the defect rather than dissolving it: |
| 71 | +the problem is not what happens AFTER a completion call, it is that the model |
| 72 | +mostly never makes one. |
| 73 | + |
| 74 | +## Root cause |
| 75 | + |
| 76 | +The private completion tool is advertised to the model as an ordinary tool. |
| 77 | + |
| 78 | +A source probe (`buildKiroPayload` with an `exec`/`wait` catalog) renders the |
| 79 | +wire tool names as `["exec","wait","codex_kiro_final_answer"]` and injects: |
| 80 | + |
| 81 | +> Valid tool names for this turn are exactly \`exec\`, \`wait\`, |
| 82 | +> \`codex_kiro_final_answer\`. These listed names are the complete top-level |
| 83 | +> tool-call surface for this turn. |
| 84 | +
|
| 85 | +That sentence comes from the shared, provider-agnostic nudge in |
| 86 | +`src/adapters/tool-catalog-nudge.ts`, which knows nothing about completion |
| 87 | +semantics. It cannot distinguish the proxy's private terminal channel from |
| 88 | +`exec`, and the same nudge closes with: |
| 89 | + |
| 90 | +> Count a tool call only after its tool result returns. |
| 91 | +
|
| 92 | +`KIRO_COMPLETION_INSTRUCTIONS` is the only text that describes the completion |
| 93 | +tool, and it never contradicts that: |
| 94 | + |
| 95 | +> When tools are available, ordinary assistant text is mid-task commentary and |
| 96 | +> does not end the turn. Continue using tools after progress updates. When the |
| 97 | +> task is fully complete and no more tool calls are needed, call |
| 98 | +> `codex_kiro_final_answer` exactly once with the complete user-facing final |
| 99 | +> answer in `answer`. Do not provide the final answer as ordinary assistant |
| 100 | +> text. |
| 101 | +
|
| 102 | +Every sentence there is about WHEN to call it. Nothing marks it as different in |
| 103 | +kind from `exec`, and nothing states what happens after. So the model holds a |
| 104 | +contract in which the terminal channel is one more ordinary tool it may defer |
| 105 | +while it keeps working — and the generic nudge's "count a tool call only after |
| 106 | +its tool result returns" applies to it as uniformly as to everything else. |
| 107 | + |
| 108 | +The failure that follows is one of SELECTION, not sequencing. Across 4069 |
| 109 | +required-mode attempts the completion tool was chosen 25 times: 0.6%. The model |
| 110 | +keeps emitting finished prose as commentary and calling ordinary tools instead |
| 111 | +of completing through the channel built for it. |
| 112 | + |
| 113 | +That is what the user sees. Measured over 1116 Kiro turns in the same two days |
| 114 | +of client rollouts: 626 turns ended through the completion channel, 462 ended |
| 115 | +on a tool call, and 28 ended with answer-shaped commentary prose and no |
| 116 | +completion call at all. Those 28 are answers the model had already finished |
| 117 | +writing — they open with "Done.", "완료", "머지까지 끝났습니다", "All ten items are |
| 118 | +done" — delivered as mid-task commentary, which by the proxy's own contract |
| 119 | +"does not end the turn". Three of them are followed by 4, 10, and 12 further |
| 120 | +tool calls after the closing summary was already on screen. |
| 121 | + |
| 122 | +The missing terminal distinction is the leading mechanism behind that measured |
| 123 | +selection failure: the terminal channel is advertised as an ordinary, deferrable |
| 124 | +tool, and nothing tells the model that this is the one call that ends the turn. |
| 125 | +It is a defect in the proxy's own injected text, not a client bug and not a |
| 126 | +stream-parsing bug. Causality is not claimed as proven — establishing it |
| 127 | +requires a live post-change comparison of the same selection rate, which this |
| 128 | +unit records as the follow-up measurement rather than asserting up front. |
| 129 | + |
| 130 | +## Fix direction |
| 131 | + |
| 132 | +State terminal semantics where the model reads them: calling the completion |
| 133 | +tool ENDS the turn, returns no tool result, and nothing may follow it. The |
| 134 | +completion tool's own schema description is the load-bearing site — it travels |
| 135 | +with the tool the nudge enumerates — with the prose contract kept consistent. |
| 136 | + |
| 137 | +Removing the tool from the enumeration is not an option: the nudge states that |
| 138 | +names mentioned only in instructions are not callable, so an unlisted |
| 139 | +completion tool would be a tool the model is told not to call. |
0 commit comments