Repository navigation
fix(transcript): settle tools stranded pending after a lost session.tool.called - #1169
Conversation
pascalandr
left a comment
There was a problem hiding this comment.
@heunghingwan The root cause checks out against the SDK reducer (session.tool.success is ignored from streaming, and the reducer's own terminal reconciliation is skipped while disconnected), and reconciling at this layer is right. Two blocking points, both from reusing scheduleAuthoritativeResync():
- P1: a round that starts during the reconciliation fetch is dropped (
opencode-data.ts:400-401). The 250 ms gate only checks past activity; once it fires,scheduleAuthoritativeResync()enters dropping mode and every event arriving during the fetch is discarded at:761-764. Repro: hold the authoritative response, then emit a new execution, an assistant start and 12 text deltas: neither the reducer nor the transcript contains the new assistant, and continued arrivals keep invalidating the fetch and prolong the dropping-mode retries. - P2: the reconciliation shrinks the display to the SDK's 20-message page without repairing pagination (
opencode-data.ts:893-895).onResyncedprojects the fresh 20-row snapshot withpreserveOmitted=falsewhile the window cursor stays put:before=200, after=20, olderCursor="after200", so loading older history then skips the 180 omitted messages.
Both go away with a reconciliation that keeps the reducer admitting events and keeps the display window's membership and cursor. #1156 (d969175b) now adds exactly that for its exhaustion path (one coalesced, non-dropping trailing read); once it lands, reusing it here would be simpler than a second path.
- P3: the "once" regression doesn't test the serial bound (
opencode-data.test.ts:1194-1234): removing both serial checks leaves both tests green, because the authoritative fixture returnsrunning, which the streaming-only detector already excludes. Return a tool stillstreaming, wait through several intervals and assert one read (it makes 3 reads without the bound). - P3:
dev-docs/TRANSCRIPT.md:15overstates the guarantee:session.tool.faileddoes settle astreamingtool, so "can never receive a terminal event" is too broad, and "work that outlives the settlement still settles in its reducer" conflicts with the wholesale replacement above. Describe the lost-success transition and the actual reconciliation.
The CI failures on this head (native-service-start, automation-forms, Rust client_state::cross_host) are unrelated to this change, and a merge with #1156 only conflicts on the adjacent constants.
529d207 to
6d86502
Compare
|
Thanks @pascalandr — all four points addressed in 6d86502, now stacked on #1156 (
Validation: 942/942 store tests and |
pascalandr
left a comment
There was a problem hiding this comment.
@heunghingwan Round 2 on 6d865028, reviewed as its delta on #1156's d969175b. Fixed: a round starting during the held fetch now stays reduced and visible (new execution, assistant start and all 12 deltas), the 200-row display keeps its membership and cursor, the serial bound lives in one place and its test fails without it (4 reads), and the doc's failed exception is right. Three blocking points remain, all where the reconciliation relies on #1156's read:
- P2: a change to the same assistant row during the fetch consumes the recovery without healing the tool (
opencode-data.ts:521-530). The row differs from its snapshot, so #1156's merge keeps the live row with its stranded tool, and the settlement serial is already spent: with another running tool in the same assistant reporting progress during the held fetch, authority saidcompletedand the result wasreads=1, strandedState=pending. Treat "the merge kept the live row over a terminal page row" as reconciliation still owed (bounded, coalesced) rather than done. - P2: the 20-row
message.syncleaves older stranded tools and running shells behind (opencode-data.ts:524-525). The display keeps its window now, but the reducer's membership becomes the SDK's latest 20 rows: a stranded tool followed by 25 newer messages stays pending (its native row is gone), and a running shell outside that page stays displayed while its latersession.shell.endedis ignored because the reducer forgot it. Reconcile the resident window that holds the target and any unfinished work (requesting 200 rows in a scratch copy made both pass), and keep unfinished reducer rows. - P2: a repair completed by #1156's trailing read never reaches the display (
opencode-data.ts:525-530). Publication hangs off the initialmessage.syncpromise only; when three destructive invalidations exhaust it, #1156's trailing read repairs the reducer on its own:reads=4, native=completed, visible=pending. This is the same gap as point 2 of my round-6 review on #1156; once that trailing read publishes, please re-check this here.
Also: dev-docs/TRANSCRIPT.md:15 still says work that outlives the settlement settles in its reducer, which the shell case in point 2 contradicts; and neither PR test covers the display preservation or the quiet gate (removing either keeps both green), so each deserves a regression.
|
Round 2 addressed in 8befef6 — the reconciliation no longer rides the SDK
Also: Validation: 944/944 store tests and |
8befef6 to
5418524
Compare
A tool row occasionally stayed "pending" forever while later tools completed and even the next round streamed, until the user switched sessions and back to force an authoritative reload. The UI's "pending" is the native reducer's tool state "streaming". A tool leaves "streaming" only through session.tool.called, and session.tool.success is reduced only from "running". When "called" is lost across a disconnect window (native events are never re-delivered; reconciliation owns recovery), the later success is silently swallowed by that guard and the row can never settle from events again. Both designed recovery paths can miss: the reducer's own settlement refresh is dropped while the connection is not "connected" (exactly around the disconnect that caused the loss), and the seeded reconnect's trailing resync defers while events keep arriving inside every one-second quiet window, so a busy session postpones it indefinitely. retireDrainedTranscript now schedules one settlement reconciliation when a drained transcript still holds a tool part in "streaming". scheduleSettlementReconcile arms a 250 ms quiet, revision-stable timer (the reconnect trailing pattern) and delegates to the existing dropping-mode scheduleAuthoritativeResync, which swaps in the persisted row and re-projects through the existing onResynced path. The stranded signature is exact: only an assistant tool part still "streaming" once the settlement drained. Work that legitimately outlives the settlement (running shells, backgrounded tools in "running") still settles in the reducer that created it, and settlementSerial bounds reconciliation to one authoritative read per settlement so a row the daemon still reports as running cannot loop fetches. Validated with two new cases in opencode-data.test.ts reproducing the lost-called sequence: the stranded part heals to completed with exactly one authoritative read, and a still-running row is read once while the reducer stays alive and the next round reduces live. All 918 store tests and tsc --noEmit pass; the pinned settlement semantics in opencode-data-settlement.test.ts and opencode-data-idle.test.ts are unchanged.
Review follow-up: the settlement reconciliation now rides the coalesced, non-dropping trailing read this branch inherits from NeuralNomadsAI#1156 instead of scheduleAuthoritativeResync. - A round that starts during the reconciliation fetch keeps admitting into the live reducer: the fetch never enters dropping mode and the read's row merge keeps streamed rows live, so the dropped-round repro from review cannot happen. - The re-projection runs with preserveOmitted on the live entry, so the display window keeps its membership and cursor; the fresh-page swap that shrank it to the SDK's 20-message page is gone. - The once-per-settlement bound lives solely in the reconcile guard; the "outlives" test now returns a still-streaming tool and asserts one authoritative read across several intervals (it makes three reads without the bound), exercising the serial check the old fixture never reached. - TRANSCRIPT.md and the detector comment describe the lost-success transition precisely: failed still settles a streaming row, so only a row whose round already ended needs the read.
Second review round: the settlement reconciliation no longer rides the SDK message.sync read. It reads one full resident window (200 rows) through the root client, seeds the authoritative page back over the reducer with unfinished live rows riding along, and publishes the projection itself. - A change to the same assistant row during the fetch can no longer consume the recovery: the fetched page is authoritative for the rows it holds, so the stranded tool's terminal state always lands. Reads superseded by admitted events (fenced on both sides of the fetch and the seed) owe one bounded, coalesced retry instead of proceeding. - The reconcile page covers the resident window, and running shells, compactions and uncompleted assistants outside it ride along, so an older stranded tool heals and a shell's later session.shell.ended still reduces in the row that created it. - Publication hangs off the reconciliation's own read: nothing waits on the SDK read's trailing recovery, so a repair always reaches the display. - Attempts are bounded per settlement (three quiet-gated reads); the quiet gate and the display-preserving projection each gained a regression test verified red without their guard, and the attempt bound test asserts reads stop once the bound is spent. TRANSCRIPT.md describes the fetched-page-wins semantics and the ride-along rows.
5418524 to
5fd9add
Compare
|
@heunghingwan Thanks for the rebase. Before another review round, I checked where the stranded row comes from on current
So
Your first regression (with and without the connected status), plus one case where a new round streams during the read and keeps its events, would cover it. Would you rework it in that direction? It should land well under 50 lines of production code. |
Reworked per review: dev's reducer already reconciles a settlement that leaves a streaming or running tool (the terminal event invalidates and re-reads messages), with two gaps — the refresh is dropped while the stream is not connected, and its result is never published, so the tool renders "pending" until the session is reloaded. finishOpenCodeDataEvent now completes that read itself: one message.sync without invalidate shares the SDK's in-flight read when connected and starts it when not, then projects the reconciled page (preserving the display window's membership) and re-runs retirement. A read superseded by events admitted after its settlement is not published — that round's own settlement reconciles instead, which keeps the pinned semantics of runtime-contract's superseded terminal check. The dedicated lane (quiet-gated timers, attempt counters, settlement serials, the 200-row scratch read and its merge) is gone; the three regressions cover the disconnected heal, the connected heal, and a new round that streams while the read is in flight (its events survive, the superseded read stays unpublished, and the next settlement heals the tool).
|
Reworked in d775f02 along exactly that line — the dedicated lane is gone (no timers, attempt counters, serials, 200-row read, One addition your sketch didn't need but The three regressions:
Verified red with the finish-read block disabled (all three fail, reproducing |
pascalandr
left a comment
There was a problem hiding this comment.
@heunghingwan Round 4 on d775f029: the smaller design works, and it is much easier to reason about. One blocking case remains.
P2: same-round events can permanently suppress the repair
packages/ui/src/stores/opencode-data.ts:913-914
The full-data revision fence treats every admitted event as a superseding round, without keeping another publication opportunity.
Reproduced through the production dispatcher:
- Start a tool without delivering
session.tool.called. - Emit
session.execution.succeededand hold its message read. - Deliver that tool's late
session.tool.success, without starting another execution. - Resolve the read with the completed authoritative row.
Result: one read, reducer completed, display still pending. The late tool event also supersedes the separate lifecycle reconciliation (session-events.ts:175-183,207), so that fallback cannot heal it either. With no later event, the row stays pending.
Please distinguish a superseding execution from same-round activity (for example, fence on a newer execution rather than on any admitted event), or keep a bounded publication obligation. A narrower scratch fence healed this repro while preserving the pinned new-execution check. TRANSCRIPT.md's publication wording should then follow the corrected fence.
Validation on this head
- 939/939 store tests pass,
tsc --noEmitclean. - Disabling the finish-read block fails all three new regressions; restoring it passes them.
- Also checked and fine: connected read sharing, disconnected warm-key read, 200-row display membership and cursor, running tool/shell retirement without repeated reads.
- Metadata/permission events also suppress publication at store level; the production lifecycle fallback healed those cases, so they are not blocking.
P0: 0 · P1: 0 · P2: 1 · P3: 0
Round-4 review: the full-data revision fence treated every admitted event as a superseding round, so a late same-round session.tool.success (the stranded tool's own terminal, swallowed by the reducer's running-status guard) permanently suppressed the publication — one read, reducer completed, display still pending, with no later event to heal it and the lifecycle fallback fenced the same way. The fence is now an execution epoch: transcripts count admitted session.execution.started events, and the settlement read publishes unless a newer execution superseded it. Same-round activity — a late terminal tool event, streamed text, metadata — no longer suppresses the repair, while the pinned superseded-new-execution check keeps its exact semantics. The in-flight-round regression asserts the read lands published with the round's events intact, and the review's repro is covered by a dedicated late-success test (verified red under the old revision fence).
|
Fixed in 003d746 — the fence is now an execution epoch rather than the full-data revision. Transcripts count admitted
|
…ence (#1187) A settlement whose terminal event fell inside a disconnect window had no trigger: finishOpenCodeDataEvent never ran, so the SDK's settlement read was never finished and a tool row stayed "pending" until a reload. A queued prompt delivered during the gap also vanished from both the queue and the transcript. The settlement read from #1169 is extracted as completeSettlementRead and now also runs from two pieces of evidence the client already gathers: the 30-second liveness pass reports each run it believed working that the final activity read no longer lists (only while the transcript still holds an unsettled tool), and the inbox sync reports an admitted item the authoritative inbox no longer lists. Neither path has a terminal event that invalidated the SDK's message key, so both invalidate it before the read; otherwise the sync engine would skip it on any reducer that synced before. The execution-epoch fence still lets only a newer execution suppress the publication, and the display window keeps its membership. Regressions start from a reducer whose first sync completed and expect exactly one read: a liveness-observed settlement heals a stranded tool, a newer execution suppresses that publication while the reducer still reconciles, an inbox sync admits the delivered prompt, the production liveness pass heals end to end, and a still-active run reads nothing. Store suites and the UI typecheck pass.
…ixes (#1192) # CodeNomad 1.0.1 - Steadier Missions, ranked history search and Linux fixes CodeNomad 1.0.1 removes the Missions project limit, makes mission states easier to read, and adds relevance-ranked search across a project's history. It also fixes transcripts and questions that could stay stale after a dropped connection, and fixes Linux AppImage display and F11 problems. ## Missions - A project is no longer limited to 20 missions, and storage limits now apply to each mission rather than the whole project. Deleted missions release their storage once their cleanup is done, and the Missions panel loads closed missions as lightweight summaries, so long mission histories stay fast (#1170, #1173, #1175, #1176, #1177, #1178, #1180). - Mission colours now follow the session list: red when an answer is needed, pulsing yellow for running work, orange for interrupted or paused missions, green for an unseen completion, and blue for a prepared mission that hasn't started. The collapsed chevron only signals answers, running work and unseen completions (#1150, #1155). - A coordinator can no longer end a one-time mission as failed on its own. When it can't continue, it asks you; a mission that stalls shows as **Interrupted** and can be resumed with **Recover interrupted mission**, including previously failed missions (#1150, #1151). - The creation form now takes up the whole Missions panel and includes the project's subagent nesting depth. An unsent draft is kept when you switch right-panel tabs or projects (#1142, #1155). - Fixed recurring missions never starting on OpenCode 2.2.0. A failed first start of a recurring run now shows a warning right away and retries after 30 seconds (#1158, #1171). - Fixed missions on OpenCode Zen's free models being refused at the coordinator's first request because of the mission session identifier format (#1133). - Fixed nested subagents stopping at depth 1 when a project config was left with an empty `experimental` block. **Inherit** now removes that empty block, and the shown depth matches what OpenCode applies (#1153). - The help now explains that a running one-time mission keeps going in OpenCode while CodeNomad is closed, and lists what waits for the backend (#1137). ## History and search - In the transcript search window, project-wide search now ranks results by relevance (rare words weigh more, exact phrases first, one result per conversation before repeats) and highlights every term. Conversation search stays chronological (#1161, #1164). - Project search scans the most recent 16,384 messages first, shows up to 40 results, and can search further back in older history (#1182). - Agents can search the project's past sessions themselves with the `session_search` Code Mode tool, which stays limited to the calling session's project (#1161, #1182). - History, cleanup and attachment views now read files stored with OpenCode 2.2.0's opt-in media blobs correctly. Before, cleanup could report a false conflict and attachments could look unavailable (#1172). ## Transcript and connection reliability - Opening a session that is already generating (after a restart, or from a subagent box) now shows its transcript and keeps streaming, instead of staying empty or frozen (#1156, #1181). - Tool rows that stayed "pending" after a missed event now settle on their own, including when the turn ended during a disconnect. A queued prompt delivered during a disconnect now appears in the transcript (#1169, #1184, #1187). - Missed questions now appear within seconds on unstable connections, such as a phone going through a tunnel. CodeNomad reconnects a stream that has gone silent, rechecks when the network comes back or the app returns to the foreground, and fetches a question the transcript shows as running when its form never arrived (#1162, #1165). - New session now shows a spinner while it is being created on a slow network. Repeated clicks or shortcuts no longer create duplicate sessions, and a failure shows an alert (#1157). - Sessions and permission requests of a repository opened as its own project no longer show up in a parent folder's project (#1174). - Session alerts, such as a usage-limit error, no longer appear a second time in the Preferences window (#1163). ## Panels and previews - When the window narrows, each panel shrinks once and then hides, and widening restores the same layouts in reverse. A panel that loses its room closes and comes back when the room returns. On phones and narrow windows, picking a session from the floating session panel closes the panel (#1144, #1149). - The active project tab can grow to 300 px so long project names fit (#1166). - Clicking a `localhost` link in a conversation opens it in the session's web preview; Ctrl+click (⌘-click on macOS) or middle-click still opens your browser. Apps that use `localStorage`, `sessionStorage` or cookies now work inside the preview (#1119). - Paired Remote Control devices can now use web previews, controlled by a new **Web previews** switch in Remote Control settings (on by default). SideCar connections now refuse requests from sandboxed pages (#1131, #1135). ## Desktop and Linux - Fixed a black window in the Linux AppImage on up-to-date systems: the AppImage now uses the system's display libraries (#1138). - Fixed F11 freezing the Linux app under X11/XWayland (#1143). - The CodeNomad server package now starts when installed with pnpm (#1186). ## Upgrade OpenCode 2.2.1 is now the recommended runtime; 2.1.2 remains the minimum, and 2.1.2, 2.1.4 and 2.2.0 keep working with an optional upgrade offer (#1188).
Summary
Occasionally a tool row in the transcript stays "pending" forever while later tools complete and even the next round streams, until the user switches sessions and back.
Why
The UI's "pending" is the native reducer's tool
state.status === "streaming". A tool leavesstreamingonly throughsession.tool.called, andsession.tool.successis reduced only fromrunning, so whencalledis lost (for example across a disconnect window) the later success is ignored. The SDK already reconciles such rows itself: on an execution terminal event with a streaming or running tool it invalidates and re-reads the session's messages, which is whatisUnsettledAtSettlementrelies on to keep the reducer alive. That read had two gaps: the SDK skips it while the stream is not connected, and nothing published its result to the display.Change
finishOpenCodeDataEvent(packages/ui/src/stores/opencode-data.ts) completes that settlement read: on an execution terminal event whose reducer still holds a streaming or running tool, it calls the SDK'smessage.syncwithout invalidating, which shares the SDK's in-flight read when connected and starts it when not, then projects the reconciled page (keeping the display window's membership and cursor) and retries retirement. Since #1156 that read completes in one fetch while the session keeps streaming and keeps every live event. A newersession.execution.startedsupersedes the publication; same-round activity, such as a late tool event, does not.Validation
opencode-data.test.ts: disconnected and connected recovery, a new step streaming during the read, and a late same-round tool success during the read; each fails without the change.tests/browser/project-tab-restore.test.ts,tests/browser/session-rendering.test.tsand the UI typecheck pass.No user-visible labels, controls, steps or defaults changed; nothing to update in
docs/help/.