Activity

1. opencode-agent commented on Oct 17, 2026

   

   opencode-agentBot

   on Oct 17, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed report and the integrity_check output.

   I couldn't reproduce this here because it depends on your storage setup. On a single host with a local disk, I ran up to 16 opencode serve (2.0.32) processes sharing one opencode.db under the same $HOME. I sent hundreds of parallel requests to them: project/location lookups, session creation, and shell commands with large output. Every request succeeded and there were no database is locked errors. The page-level corruption needs writers on different hosts sharing one database file over NFS/WekaFS, which I couldn't set up. So this needs someone who has a shared-filesystem or cluster environment.

   On the latest dev, the database is still always opened in WAL mode with a 5s busy_timeout. There is no network-filesystem detection and no option to turn WAL off. Until that changes, giving each node or job its own data directory (for example by setting XDG_DATA_HOME to a node-local path) should avoid sharing one SQLite file across hosts.

   Related issues, most relevant first:

   • SQLite database corruption (database disk image is malformed) when running concurrent sessions on NFS #14970: SQLite corruption on NFS with concurrent sessions (v1, closed)
   • OpenCode fails when two hosts share an NFS home directory #43620: two hosts sharing an NFS home (v1, closed)
   • fix(core): retry sqlite statements on lock timeout #54396: open PR to retry SQLite statements on lock timeout
   • session: two server processes sharing one opencode.db allocate session_message.seq independently - UNIQUE(seq) collisions fail sessions #53146 / Cross-process session concurrency: two processes can run loops on the same session simultaneously #49367: several server processes sharing one opencode.db
   • core: concurrent first start on a new opencode.db fails with "database is locked" / CREATE TABLE error #53879: concurrent first start on a new opencode.db fails with "database is locked"

   It would help to know whether the lock storms also happen when every opencode process runs on one node, or only when agents on several nodes share the same $HOME.
2. mvillmow commented on Oct 17, 2026

   

   mvillmow

   on Oct 17, 2026

   Author

   More actions

   Follow-up: third failure mode found — cross-node serve --service registration teardown on shared $HOME

   After posting, we diagnosed a further interaction in the same environment (same setup: opencode v2.0.32, DB in WAL on a parallel network FS, 10+ agents plus sub-agents in tmux/Slurm). Adding it here because it is likely the same root cause (a single per-user coordination file on shared $HOME used by many processes across hosts) and because its symptoms (servers vanishing / tmux client disconnects) overlap with the lock storm.

   What happens

   opencode serve --service maintains ~/.local/state/opencode/service.json. With $HOME shared across cluster nodes, any node's serve --service sees the previous registration (different PID, possibly a server running on a different host), treats it as conflicting, shuts down the previous server, and re-registers itself ("last writer wins"):

   level=WARN message="managed service registration replaced; shutting down"
     serviceID=... servicePID=<node A pid>
     registration=<shared-home>/.local/state/opencode/service.json
     observedServiceID=... observedServicePID=<new pid>
   

   Observed loop, within minutes, repeatedly: server A (node X) running fine with agents → someone on node Y (re)starts serve --service → A shuts down → B registers on node Y → later, a (re)start on node X does the same → B shuts down, unrelated to load. Meanwhile at the time that A's registration was torn down, there was no OOM kill and no crash trace in stderr; the only death mechanism is this registration swap. Clients (tmux sessions on any node that trusts the shared service.json) then attempt to connect to the stale registration URL (which on the other nodes points at that node's local 127.0.0.1... i.e., unreachable) and appear as silent disconnects.

   We also confirmed this is completely independent of the DB integrity: in a window where agents ran a full parallel swarm (thousands of spawned child commands registering tool calls and DB writes; DB grew ~12 MB and the WAL was ~10 MB), a post-swarm PRAGMA integrity_check (full, not quick_check) on the repaired DB read ok and .db-wal checkpointing was clean. The DB has remained healthy since the .recover repair — the standby branch of the lock/ordering problems is still the one tearing tmux sessions down. Worse, the advertised url is http://127.0.0.1:<port> — a URL that is only meaningful on the node that wrote it. Clients on other nodes reading the shared service.json therefore point at their own loopback port, where either nothing listens (silent connection refusal) or an unrelated service listens.

   Minimal reproduction (per the Suggested-fix section above)

   1. Two nodes (or two cluster shells). Shared $HOME in between.
   2. Node A: opencode serve --service & → registers.
   3. Node B: opencode serve --service & → logs "managed service registration replaced; shutting down", and A's server goes away.
   4. Node A clients break; service.json now contains node B's PID (a number that does not exist on node A) and a 127.0.0.1 URL.

   Suggested fix directions

   • Make the service registration file host-name- or node-scoped (e.g. keyed by hostname/machine-id, or write it under a per-host state directory instead of one shared file). service.json merely advertises a URL, so the design intent of "one service per user, known by the last writer" only holds where each user maps to exactly one host — which shared $HOME defeats.
   • If serve --service starts and observes an unrelated registration whose URL points at a different host (or a PID that is not scan-able via kill(pid, 0) or /proc/<pid> owning the expected process name), it should NOT kill that server but should fall back to starting a fresh server bound to a node-scoped state entry (or to per-node port).
   • The last-writer-wins teardown should produce a loud terminal message (which is what does happen: "managed service registration replaced; shutting down") plus, ideally, a probe: GET /api/info on the observed URL would return UnauthorizedError without auth but proves whether the observed URL is unsatisfied.

   Context to the concerns from the main report: the SQLite page-corruption story is still the DB root cause observed on this shared filesystem (recovered cleanly again more than once now), but the servers vanishing mid-swarm appears to be primarily this registration liveness issue rather than DB/memory crashes. Both are worth investigating independently.
3. bull-rainer-thierfelder commented on Nov 13, 2026

   

   bull-rainer-thierfelder

   on Nov 13, 2026

   More actions

   v2.3.0 data point: single-node corruption during service handoff, and proof the WAL pragma is set unconditionally on every open

   Environment: opencode v2.3.0 (release, channel latest), single-user desktop-grade workload (no agent swarms), $HOME on an NFSv4 automount; pooled/random Linux hosts, so the database must stay on NFS.

   1. Corruption happened on a single node — no multi-host writers, no agent swarm. During a version-upgrade handoff, two opencode serve --service generations overlapped (log: managed service registration replaced; shutting down twice, plus EBUSY: resource busy or locked errors removing files under ~/.local/state/opencode/locks/). The next generation's very first models.dev access failed:

   EffectDrizzleQueryError: Failed query: select "value" from "kv" where "kv"."key" = ?
   cause: SQLiteError: database disk image is malformed
   

   PRAGMA integrity_check (on a copy) showed exactly one damaged tree — Tree 15 page 15: btreeInitPage() returns error code 11, tree 15 being the kv table (rootpage 15); all other tables passed. So the single-node handoff window alone sufficed to corrupt the WAL database; cross-node access is not required. To your question in the first comment: our case is single-node only, corruption but no lock storm (light write load).

   Recovery: service stop → sqlite3 <db> ".recover" → integrity-check → atomic swap → restart. Every readable row survived (sessions, events, migrations); only the KV cache rows were lost, and the plugin refetched its catalog. Kept the corrupt file for forensics.

   2. Empirical proof for the suggested "documented override" — a file-level journal-mode switch does not stick. With the service stopped we flipped the database to rollback-journal mode (PRAGMA journal_mode=DELETE, file header write/read version bytes verified 01 01). The first server start reverted it within seconds (header 02 02, -wal/-shm recreated). The reason is visible in current main — packages/core/src/database/database.ts runs unconditionally for file databases:

   if (supportsTuningPragmas && !durableObject) {
     yield* db.run("PRAGMA journal_mode = WAL")
     yield* db.run("PRAGMA synchronous = NORMAL")
     yield* db.run("PRAGMA busy_timeout = 5000")
     ...
   }

   So in the current release, journal mode cannot be opted out of by any user-visible mechanism; the override has to exist in OpenCode itself. Given bun:sqlite leaves WAL opt-in, an env/config knob (e.g. OPENCODE_DB_JOURNAL=delete) plus a warning when the DB path is on a network filesystem would cover environments like ours (pooled hosts, NFS-only homes, XDG_DATA_HOME on local disk is not an option because nothing host-local persists).

   Meanwhile our mitigations: opencode service stop before version upgrades (the handoff window is the danger), never running sessions from two hosts simultaneously, periodic DB+WAL snapshots, and a scripted .recover-swap repair for when it recurs.