Skip to content

fix(opencode): run one session loop at a time across processes - #50968

Closed
iceteaSA wants to merge 1 commit into
anomalyco:devfrom
iceteaSA:session-run-lease
Closed

iceteaSA wants to merge 1 commit into
anomalyco:devfrom
iceteaSA:session-run-lease

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Sep 23, 2026 •

Copy link
Copy Markdown

Issue for this PR

Fixes #49367 (reported by @Duong-Truong-Anh).

Type of change

  • Bug fix

What does this PR do?

Two opencode processes on the same project can run the loop for the same session at the same time. SessionRunState keeps its runners in a per-process map, so ensureRunning only joins a run in its own process. Each loop's history reload sees the other loop's unfinished assistant message, the exit check fails, and the two take turns answering the same user message. On my machine this hit four sessions, each with 15 to 45 consecutive finish: "stop" replies to one message. Every time, a second TUI or CLI had been opened on the same project.

This adds a per-session lock shared across processes, built on EffectFlock (a mkdir lock with a heartbeat and stale-lock takeover):

  • EffectFlock gains tryAcquire (one attempt, including the stale-lock break, registered for release in the same uninterruptible step), Held.verify (is the lock still mine?) and holder (who holds it, for logging). Breaking a stale lock now logs the previous holder's pid.
  • The existing heartbeat never moved: fs.utimes(path, new Date(), new Date()) piped into Effect.repeat reruns one effect whose dates were fixed when it was built, so every lock looked abandoned 60 s after it was taken. Both acquire and tryAcquire now build the call per tick with Effect.suspend. The acquire half is also in fix(core): refresh EffectFlock heartbeats on every tick #51181 on its own, since it affects upstream locks (npm install, repository clone) without this PR.
  • SessionRunState wraps the work of ensureRunning and startShell in the lock. A process that can't get it marks the session busy and retries with a short jittered sleep that can be interrupted, so cancel and shutdown stop the wait at once. After 60 seconds it logs a warning with the holder's pid and hostname, and repeats every 5 minutes.
  • runLoop checks owns(sessionID) at the top of every iteration. If another process took the lock over (the holder stalled for more than the 60 second stale window), the loop fails with SessionRunState.LeaseLostError before its next provider call, instead of returning the last assistant message: that message can be the other process's unfinished turn, and a background task reported it as a completed, empty result. The HTTP handlers and the task tool report it as an error. The holder's in-flight turn can still finish, tool calls included, so this is best-effort recovery, not fencing. With a live heartbeat the two never overlap.

A second process's prompt is answered either by the running process at its next turn boundary or by the second process once it gets the lock. That covers the window a stricter exit check can't: a prompt stored after the running loop's final exit check, which nothing would otherwise answer.

Out of scope: aborting a run from a different process (abort in the waiting process cancels only its own wait), the V2 SessionRunCoordinator, and #43133, where run --continue picks a session another process is using.

Two opencode run changes came out of testing:

  • A waiting run whose prompt the other process answered printed nothing, because /event only carries the local process's events. After the prompt returns, run now fetches the assistant messages that answer its prompt and renders them through the same part renderer the event loop uses, skipping parts it already printed. Text and --format json output (including step_start/step_finish) match a local run, multi-turn replies included. --attach is unchanged.
  • opencode run --format json could print two error records for one rejected prompt, one from the session.error event and one from the HTTP rejection. The server publishes session.error exactly once with or without this change; before, the process usually exited before the event reached the CLI, and the lock's small delay let it arrive first. The CLI now drops the request error when a session error was already printed for the run, and drops a session error only when it repeats a request error with the same name and message. A different second error is still printed.

How did you verify your code works?

  • The cross-process tests build two independent layer graphs (separate memo maps and scopes) over one database file and one lock directory, and drive real prompt.loop runs against the test LLM. They check that the two never run at once, that each user message gets exactly one reply (by parentID), that a prompt stored while the other process holds the lock is answered once after release, that a cancelled wait returns within a second without touching the holder's lock, and that a holder whose lock is taken over mid-turn or mid-tool stops before its next turn, with the tool result on its own message.
  • A heartbeat test backdates held locks (both acquire and tryAcquire) past the stale window, waits one interval, and checks the heartbeat is fresh and a second attempt can't take the lock. It fails on the old heartbeat. A lost lease is asserted as LeaseLostError in the takeover tests and as an error frame for a background task.
  • Two tests in run-process.test.ts spawn two real opencode run processes on one database and hold the first one's model reply, so the first process answers the second one's prompt. The second prints that reply exactly once, with the same JSON shape as a local run, for single-turn and multi-turn replies. Without the fix it printed nothing.
  • Removing the lock, making owns always true, dropping the busy status while waiting, or making the wait uninterruptible each fails a test. So do narrowing the CLI deduper back to "drop every second error", rendering only the last reply message, and dropping the already-printed check.
  • packages/opencode: 3,594 pass, 2 fail. Both failures (shell captures stdout and stderr in completed tool output and mcp session recovery > reinitializes and retries once after a session-bound POST returns 404) hit their time limit during the full run on a loaded machine and pass when run alone. packages/core: 1,102 pass; the one failure, Ripgrep > never includes git metadata, also fails on clean dev. Typecheck clean.
  • The heartbeat bug was found live after the first version shipped in a local build: with several processes on one machine, a second process took over a session 66 s into another process's run and both ran it at once. Every held lock's heartbeat mtime equalled its creation time.
  • Live, with an installed build: two opencode run --session <id> processes against one session, the second prompting 4 s after the first. Each user message got exactly one reply, and no two replies overlapped in time. The first process answered the second prompt at its next turn boundary, and the second made no model call and exited once the lock was released. That run predates the output fix above, which is how the silent second run showed up.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

Two opencode processes on one project could each run the loop for the same session, because SessionRunState keeps its runners in a per-process map. Each loop's reload saw the other's unfinished assistant message, the exit check failed, and they took turns answering one user message.

A per-session EffectFlock lock now wraps the work of ensureRunning and startShell. A process that can't take it marks the session busy and retries with an interruptible jittered sleep, warning with the holder's pid after 60 s. The holder refreshes the lock's heartbeat every 20 s. runLoop checks it still owns the lock before every iteration; if another process took it over (the holder stalled past the 60 s staleness window), the loop fails with SessionRunState.LeaseLostError instead of returning the last assistant message, so callers such as the task tool report an error rather than an empty completion. A stalled holder's in-flight turn may still finish, so this is best-effort, not fencing. Breaking a stale lock logs the previous holder's pid.

EffectFlock gains tryAcquire (one attempt, release registered atomically), Held.verify and holder. It also fixes the heartbeat of acquire: the repeated utimes call was built once, so every tick re-applied the acquisition time and a lock held past 60 s looked abandoned.

opencode run: a waiting run whose prompt the other process answered used to print nothing, because /event only carries the local process's events. After the prompt returns, run now renders the reply's messages through the same part renderer as the event loop, skipping parts it already printed, so text and JSON output match a local run. It also prints one error record per rejected prompt instead of reporting the same failure from both the session event and the HTTP rejection.

Fixes anomalyco#49367
@thdxr

thdxr commented Oct 10, 2026

Copy link
Copy Markdown
Member

This branch has been upgraded to V2. We're no longer taking PRs for V1. If you think this is still relevant, please port it over and open it for V2.

— from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cross-process session concurrency: two processes can run loops on the same session simultaneously

2 participants