Skip to content

Commit 0f52a9b

Browse files
committed
fix(sync): finalize tool parts orphaned by an interrupted turn after settlement
When a managed OpenCode process dies mid-turn (crash, health-check restart), the persisted turn never settles: the trailing assistant message has no time.completed and its tool parts stay pending/running forever — the server never finalizes them (anomalyco/opencode#19023). The existing settle-triggered tail refresh refetches the same stale records, so the UI kept running tool timers and working styling indefinitely (#2577). Now, when a session is authoritatively settled (session.idle/ session.error event, or an authoritative status snapshot lowering a previously busy session) and the trailing assistant message is still unfinished with active tool parts and no pending question/permission, the orphaned parts are finalized locally as error/"Interrupted" with an end time — the same shape OpenCode itself writes for cancelled tools. The mark is gated on an explicit idle status (absent status is "unknown", never judged), never applies while busy (including question/permission waits), and a later terminal event or refresh supersedes it while a stale running refresh cannot regress it (the reducer and materializer already preserve final statuses). Fixes #2577
1 parent 1fd80b9 commit 0f52a9b

4 files changed

Lines changed: 287 additions & 0 deletions

File tree

‎packages/ui/src/sync/DOCUMENTATION.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,8 @@ Incomplete-session materialization is deduplicated by runtime, directory, and se
204204

205205
When `session.idle` or `session.error` settles a session but the trailing assistant message still contains a `pending` or `running` tool, sync refreshes that session tail. This narrowly reconciles a missed terminal tool-part event without refetching normally completed turns or stale tools from older turns. A stale refresh or delayed part event cannot regress a locally observed terminal tool to an active status.
206206

207+
When a session is authoritatively settled — `session.idle`/`session.error` event, or an authoritative status snapshot that lowers a previously busy session — and the trailing assistant message is still *unfinished* (`time.completed` missing) with active tool parts and no pending question/permission, the turn is treated as interrupted (managed OpenCode process died mid-turn; the server never finalizes the parts, see openchamber#2577 / anomalyco/opencode#19023). The active parts are finalized locally as `error`/`Interrupted` with an end time, so tool timers stop and cards render the error state. The mark is gated on an explicit idle status (absent status is "unknown", never judged), never applies while the session is busy (including question/permission waits), and a later terminal event or refresh supersedes it while a stale `running` refresh cannot regress it.
208+
207209
Directory stores also own session-keyed sidecar notification channels for permissions and message materialization. High-frequency realtime part events annotate the exact session/message before committing, so visible records, user history, renderability, and sidebar permission rows are not notified by unrelated sessions. Structural message replacements notify only changed subscribed session buckets; unannotated bulk part replacement conservatively resets active message subscribers so bootstrap, pagination, rollback, and legacy writers cannot leave stale projections.
208210

209211
Message sidecar consumers also filter targeted updates by purpose before notifying React. Suspended live-tail text/reasoning changes do not rebuild visible message records, but structural Task session identity changes bypass suspension so a parent can link a newly created subagent immediately. Assistant-only part changes do not rebuild user input history, and targeted updates that preserve authoritative part buckets do not recheck a session that is already renderable. Message replacements, removed final part buckets, and conservative resets always notify.
Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
/**
2+
* Tests for interrupted-turn reconciliation (#2577): when a managed OpenCode
3+
* process dies mid-turn, the persisted turn never settles — the trailing
4+
* assistant message has no time.completed and its tool parts stay running.
5+
* Once the session is authoritatively settled, `interruptedTurnToolParts`
6+
* finalizes the orphaned parts locally.
7+
*/
8+
import { describe, expect, test } from "bun:test"
9+
import type { Message, Part } from "@opencode-ai/sdk/v2/client"
10+
import { interruptedTurnToolParts } from "../sync-context"
11+
import type { DirectoryStore } from "../child-store"
12+
import { INITIAL_STATE } from "../types"
13+
14+
function state(overrides: Partial<DirectoryStore> = {}): DirectoryStore {
15+
return {
16+
...INITIAL_STATE,
17+
session_status: {},
18+
message: {},
19+
part: {},
20+
question: {},
21+
permission: {},
22+
...overrides,
23+
} as unknown as DirectoryStore
24+
}
25+
26+
function runningTool(id: string, messageID: string, start = 1000): Part {
27+
return {
28+
id,
29+
messageID,
30+
sessionID: "ses_1",
31+
type: "tool",
32+
tool: "bash",
33+
state: { status: "running", time: { start }, input: {} },
34+
} as unknown as Part
35+
}
36+
37+
function completedTool(id: string, messageID: string): Part {
38+
return {
39+
id,
40+
messageID,
41+
sessionID: "ses_1",
42+
type: "tool",
43+
tool: "bash",
44+
state: { status: "completed", time: { start: 1000, end: 2000 }, input: {} },
45+
} as unknown as Part
46+
}
47+
48+
function pendingTool(id: string, messageID: string): Part {
49+
return {
50+
id,
51+
messageID,
52+
sessionID: "ses_1",
53+
type: "tool",
54+
tool: "bash",
55+
state: { status: "pending", time: { start: 1000 }, input: {} },
56+
} as unknown as Part
57+
}
58+
59+
function unfinishedAssistantMessage(id: string): Message {
60+
return { id, sessionID: "ses_1", role: "assistant", parentID: "", modelID: "", providerID: "", mode: "primary", system: "", agent: "", model: "", time: { created: 10 } } as unknown as Message
61+
}
62+
63+
function finishedAssistantMessage(id: string): Message {
64+
return { id, sessionID: "ses_1", role: "assistant", parentID: "", modelID: "", providerID: "", mode: "primary", system: "", agent: "", model: "", time: { created: 10, completed: 2000 } } as unknown as Message
65+
}
66+
67+
describe("interruptedTurnToolParts (#2577)", () => {
68+
test("settled session with unfinished message and running tool finalizes the part", () => {
69+
const store = state({
70+
session_status: { ses_1: { type: "idle" } },
71+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
72+
part: { msg_1: [runningTool("tool_1", "msg_1")] },
73+
})
74+
75+
const result = interruptedTurnToolParts(store, "ses_1", 5000)
76+
expect(result).not.toBeNull()
77+
const part = result!.parts[0] as { state: { status: string; error: string; time: { end: number } } }
78+
expect(part.state.status).toBe("error")
79+
expect(part.state.error).toBe("Interrupted")
80+
expect(part.state.time.end).toBe(5000)
81+
})
82+
83+
test("busy session is never marked (live work)", () => {
84+
const store = state({
85+
session_status: { ses_1: { type: "busy" } },
86+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
87+
part: { msg_1: [runningTool("tool_1", "msg_1")] },
88+
})
89+
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
90+
})
91+
92+
test("absent status is unknown, not settled — never marked", () => {
93+
const store = state({
94+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
95+
part: { msg_1: [runningTool("tool_1", "msg_1")] },
96+
})
97+
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
98+
})
99+
100+
test("finished message is not an interruption (tail refresh reconciles it)", () => {
101+
const store = state({
102+
session_status: { ses_1: { type: "idle" } },
103+
message: { ses_1: [finishedAssistantMessage("msg_1")] },
104+
part: { msg_1: [runningTool("tool_1", "msg_1")] },
105+
})
106+
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
107+
})
108+
109+
test("pending question means the turn is waiting for input, not interrupted", () => {
110+
const store = state({
111+
session_status: { ses_1: { type: "idle" } },
112+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
113+
part: { msg_1: [runningTool("tool_1", "msg_1")] },
114+
question: { ses_1: [{ id: "q_1", sessionID: "ses_1", questions: [{ question: "?", header: "h", options: [{ label: "a", description: "" }] }] }] },
115+
})
116+
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
117+
})
118+
119+
test("pending permission means the turn is waiting for input, not interrupted", () => {
120+
const store = state({
121+
session_status: { ses_1: { type: "idle" } },
122+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
123+
part: { msg_1: [runningTool("tool_1", "msg_1")] },
124+
permission: { ses_1: [{ id: "p_1", sessionID: "ses_1", permission: "bash", patterns: [], metadata: {}, always: [] }] },
125+
})
126+
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
127+
})
128+
129+
test("only active parts are finalized; completed parts are untouched", () => {
130+
const store = state({
131+
session_status: { ses_1: { type: "idle" } },
132+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
133+
part: {
134+
msg_1: [runningTool("tool_1", "msg_1"), completedTool("tool_2", "msg_1"), pendingTool("tool_3", "msg_1")],
135+
},
136+
})
137+
138+
const result = interruptedTurnToolParts(store, "ses_1", 5000)
139+
expect(result).not.toBeNull()
140+
const statuses = result!.parts.map((part) => (part as { state: { status: string } }).state.status)
141+
expect(statuses).toEqual(["error", "completed", "error"])
142+
})
143+
144+
test("no active parts → no change", () => {
145+
const store = state({
146+
session_status: { ses_1: { type: "idle" } },
147+
message: { ses_1: [unfinishedAssistantMessage("msg_1")] },
148+
part: { msg_1: [completedTool("tool_2", "msg_1")] },
149+
})
150+
expect(interruptedTurnToolParts(store, "ses_1")).toBeNull()
151+
})
152+
})

‎packages/ui/src/sync/__tests__/materialization.test.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,39 @@ describe("materializeSessionSnapshots", () => {
167167
expect(mergedPart.state?.time?.end).toBe(2000)
168168
})
169169

170+
test("does not regress a locally interrupted tool (error + end) when a stale running snapshot arrives", () => {
171+
// The #2577 mark writes status "error" + end time; a later stale refresh
172+
// that still reports the part as running must not undo it.
173+
const interruptedTool = {
174+
id: "prt_1",
175+
messageID: "msg_1",
176+
sessionID: "ses_1",
177+
type: "tool",
178+
state: { status: "error", error: "Interrupted", time: { start: 1000, end: 5000 } },
179+
} as unknown as Part
180+
const staleRunningTool = {
181+
id: "prt_1",
182+
messageID: "msg_1",
183+
sessionID: "ses_1",
184+
type: "tool",
185+
state: { status: "running", time: { start: 1000 } },
186+
} as unknown as Part
187+
const state = {
188+
message: { ses_1: [message("msg_1")] },
189+
part: { msg_1: [interruptedTool] },
190+
}
191+
192+
const result = materializeSessionSnapshots(
193+
state,
194+
"ses_1",
195+
[{ info: message("msg_1"), parts: [staleRunningTool] }],
196+
)
197+
198+
expect(result.part.msg_1[0]).toBe(interruptedTool)
199+
expect(result.part.msg_1[0]).not.toBe(staleRunningTool)
200+
expect((result.part.msg_1[0] as { state: { status: string } }).state.status).toBe("error")
201+
})
202+
170203
test("does not regress a completed tool when a stale running snapshot arrives", () => {
171204
const completedTool = {
172205
id: "prt_1",

‎packages/ui/src/sync/sync-context.tsx‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -628,6 +628,19 @@ async function resyncDirectorySessionStatuses(
628628
applySessionStatusSnapshot(store, nextStatuses, candidateSessionIds, mode)
629629
if (mode === "authoritative") {
630630
applyGlobalSessionStatusSnapshot(directory, nextStatuses, candidateSessionIds)
631+
// An authoritative snapshot that settles sessions previously observed
632+
// busy/retry can orphan running tool parts (managed process died
633+
// mid-turn, #2577): finalize them now. The snapshot write above already
634+
// lowered their status to explicit idle, which is the gate the helper
635+
// requires — a session the snapshot reports busy stays untouched.
636+
for (const sessionId of candidateSessionIds) {
637+
const interrupted = interruptedTurnToolParts(store.getState(), sessionId)
638+
if (interrupted) {
639+
store.setState((state) => ({
640+
part: { ...state.part, [interrupted.messageID]: interrupted.parts },
641+
}))
642+
}
643+
}
631644
}
632645
return nextStatuses
633646
}
@@ -1758,11 +1771,98 @@ function handleEvent(
17581771
messageID,
17591772
})
17601773
}
1774+
// The reducer already wrote the idle/error status into `draft`; mark the
1775+
// orphaned tools using the batched state and publish through the batch.
1776+
if (sessionID) {
1777+
const interrupted = interruptedTurnToolParts(state, sessionID)
1778+
if (interrupted) {
1779+
cloneField("part", (value) => ({ ...(value ?? {}) }))
1780+
;(draft as DirectoryStore).part[interrupted.messageID] = interrupted.parts
1781+
if (batch) {
1782+
batch.states.set(store, draft as DirectoryStore)
1783+
batch.changedStores.add(store)
1784+
} else {
1785+
store.setState({ part: { ...(store.getState().part), [interrupted.messageID]: interrupted.parts } })
1786+
}
1787+
}
1788+
}
17611789
}
17621790

17631791
updateRoutingIndexFromEvent(routingIndex, resolvedDirectory, payload)
17641792
}
17651793

1794+
// ---------------------------------------------------------------------------
1795+
// Interrupted-turn reconciliation
1796+
//
1797+
// A managed OpenCode process can die mid-turn (crash, health-check restart).
1798+
// The persisted turn then never settles: the trailing assistant message has
1799+
// no `time.completed` and its tool parts stay `pending`/`running` forever —
1800+
// the server never finalizes them (anomalyco/opencode#19023). The
1801+
// settle-triggered tail refresh above refetches the same stale records, so
1802+
// the UI would keep running tool timers and "working" styling indefinitely
1803+
// (#2577).
1804+
//
1805+
// OpenCode keeps a turn's session busy while it is genuinely alive —
1806+
// including while waiting for a question/permission reply — so once a
1807+
// session is AUTHORITATIVELY settled (a `session.idle`/`session.error`
1808+
// event, or an authoritative status snapshot that lowers a previously busy
1809+
// session) and the trailing assistant message is still unfinished with
1810+
// active tool parts and no pending question/permission, the turn is
1811+
// definitively interrupted. Finalize the orphaned parts locally as
1812+
// `error`/`Interrupted` with an end time — the same shape OpenCode itself
1813+
// writes for cancelled tools. A later terminal part event or a refresh that
1814+
// carries the true terminal state supersedes the mark; a stale refresh that
1815+
// still reports `running` is rejected by the reducer's and the materializer's
1816+
// final-status preservation.
1817+
export function interruptedTurnToolParts(
1818+
state: DirectoryStore,
1819+
sessionID: string,
1820+
now = Date.now(),
1821+
): { messageID: string; parts: Part[] } | null {
1822+
if ((state.question?.[sessionID] ?? []).length > 0) return null
1823+
if ((state.permission?.[sessionID] ?? []).length > 0) return null
1824+
1825+
const status = state.session_status?.[sessionID]
1826+
if (!status || status.type !== "idle") {
1827+
// Absent status is "unknown", not settled (the reducer maps both
1828+
// session.idle and session.error to {type:"idle"}): never judge an
1829+
// interrupted turn without an authoritative settle signal.
1830+
return null
1831+
}
1832+
1833+
const messageID = getStaleRunningToolMessageID(state, sessionID)
1834+
if (!messageID) return null
1835+
const message = (state.message[sessionID] ?? []).find((candidate) => candidate.id === messageID)
1836+
if (!message) return null
1837+
if (typeof (message as { time?: { completed?: unknown } }).time?.completed === "number") {
1838+
// The turn finished; a missed terminal tool event is the tail refresh's
1839+
// job, not an interruption.
1840+
return null
1841+
}
1842+
1843+
const current = state.part[messageID]
1844+
if (!current) return null
1845+
1846+
let changed = false
1847+
const nextParts = current.map((part) => {
1848+
if (part.type !== "tool") return part
1849+
const partState = (part as { state?: { status?: unknown; time?: { start?: number } } }).state
1850+
if (!partState) return part
1851+
if (partState.status !== "pending" && partState.status !== "running") return part
1852+
changed = true
1853+
return {
1854+
...part,
1855+
state: {
1856+
...partState,
1857+
status: "error",
1858+
error: "Interrupted",
1859+
time: { ...(partState.time ?? {}), end: now },
1860+
},
1861+
} as Part
1862+
})
1863+
return changed ? { messageID, parts: nextParts } : null
1864+
}
1865+
17661866
// ---------------------------------------------------------------------------
17671867
// Provider
17681868
// ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)