session: two server processes sharing one opencode.db allocate session_message.seq independently - UNIQUE(seq) collisions fail sessions #53146

Activity

1. github-actions

   assigned

   rekram1-node

   on Oct 4, 2026
2. github-actions

   added

   needs:complianceThis means the issue will auto-close after 2 hours.This means the issue will auto-close after 2 hours.

   on Oct 4, 2026
3. ImBIOS commented on Oct 4, 2026

   

   ImBIOS

   on Oct 4, 2026

   Contributor

   More actions

   Matching topology seen from the T3-Code side, plus an event-table variant of the same collision.

   Process topology sharing one WAL-mode opencode.db (~9.6G):

   • opencode serve 127.0.0.1:36681 (T3-managed, ran the original 13-minute streaming turn),
   • opencode serve 127.0.0.1:41105 (T3-managed, spawned ~1 min after the first crash; the retry ran here),
   • opencode serve --service (not T3-owned, started mid-turn).

   Variant: our collision was on the legacy event/event_sequence tables rather than session_message: the allocator read a stale counter (691 vs committed max 728) and the overlapping insert aborted the entire drain (prompt_async failed), bricking the session for plain retries — not just a single failed turn.

   This suggests the fix directions (DB-derived allocation, retry-on-UniqueViolation, single-writer guard) should cover the event surface too, not only session_message. Happy to provide timestamps/row dumps if it helps repro.
4. github-actions

   removed

   needs:complianceThis means the issue will auto-close after 2 hours.This means the issue will auto-close after 2 hours.

   on Oct 4, 2026
5. github-actions commented on Oct 4, 2026

   

   github-actionsBot

   on Oct 4, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for updating this issue with the Bug Report template and reproduction details.
6. ImBIOS commented on Oct 5, 2026

   

   ImBIOS

   on Oct 5, 2026

   Contributor

   More actions

   Follow-up with a next-day occurrence on the same session, now on this exact table — and with only ONE live server.

   Setup: all old serves were gone; a single fresh opencode serve (started 5 min before the retry) loaded the session. Committed head in session_message is seq 1118 (101 sparse rows). The drain wrote system@731 and user@732 (both gaps, committed fine), then died in SessionRunner.publishLLMEvent inserting the first assistant message at seq 733 — occupied since the previous day. The outcome-bookkeeping insert (idle) at 734 collided as well. Caller (T3) still shows the run as running with no further events: wedged, not just failed.

   So a single fresh server process still initializes the session allocator from a stale view (~731) instead of the committed head (1118) and replays overlapping seq space. Session load reconciling with max(seq) (direction 3) would have survived this; nothing else would. Happy to provide the conflicting row ids/timestamps.
7. satwareAG-ironMike commented on Oct 5, 2026

   

   satwareAG-ironMike

   on Oct 5, 2026

   Author

   More actions

   New data point (2026-10-05): single writer also rewinds - and the stale counter survives a server restart

   Follow-up from satware AG (host: mw workstation, opencode 2.0.23). A second incident the next day shows the dual-process topology is not required for this failure mode.

   What happened

   Session ses_ef60a5c98ffe85WeCqrI5WMuhq: ~60 healthy turns 04:47-05:58 local (seqs advanced 5 -> 821, all turns finish=stop). At 05:58 the agent ran its project EOD routine from inside the session, whose teardown performs chunked DELETEs + wal_checkpoint(TRUNCATE) on the live opencode.db (single server process, verified via lsof - no second holder at any point that day).

   At 06:03:49 the next assistant message persisted at seq 2 - the counter was re-initialized from a near-empty view, the same signature as the second stream in the original report. From then on every turn collided:

   ERROR "Failed to drain Session" ... UNIQUE constraint failed: session_message.session_id, session_message.seq
     insert params: msg_10a3ca289001..., assistant, seq 35, {"content":[]}
   

   Logged collisions: seq 35 and 138 (server process A), then seq 652 and 688 (server process B). Each crash ends the turn idle outcome=failed; the retry re-generates the same step, so the session visibly loops (55 iterations, ~1.9 M input tokens). Queued user prompts - including an explicit "stop looping" - cannot break the cycle because the loop is the persist-fail/retry cycle below the model layer. Only a hard interrupt stopped it.

   Persistence across restart is the aggravating part

   Process A's watchers stopped at 05:43:20Z and a fresh opencode serve --service (B) started the same second - clean handover, no overlap (single lsof holder throughout). B still collided at 652/688, marching up through seq space already occupied by A's healthy rows. So the per-session counter is not re-seeded from MAX(seq) on restart; poisoned state survives process replacement, making the session permanently unresumable.

   Ask

   1. Seed per-session session_message.seq allocation from MAX(seq) of persisted rows (or a persisted counter row) at process start and after any cache invalidation - the "near-empty view" initialization is the common root of both incidents.
   2. Fail fast: on UNIQUE(seq) violation during publish, abort the turn as failed instead of retrying the step - the retry regenerates identical content and turns one bad write into an infinite loop.
   3. Hard cap on consecutive identical-step retries per turn would have bounded this to ~1 iteration instead of 55.

   Cross-ref: internal RCA satware.ai/harness#1166 (comment 32270 has the full evidence chain).
8. nkoynov commented on Oct 7, 2026

   

   nkoynov

   on Oct 7, 2026

   More actions

   In our case (T3 + a CLI-started managed service, 2.0.22) the trigger was #51213's sweep; seqs did not collide because durable seqs come from event_sequence inside the write transaction, but every resumed step ran on both servers.
9. Seamus05 commented on Oct 19, 2026

   

   Seamus05

   on Oct 19, 2026

   More actions

   Cross-reference with single-writer evidence on 2.0.31 — no second process needed, and the wedge is decided at reload, not at the write.

   Data from a 1.3 GB opencode.db (WAL, integrity_check ok, 2167 sessions):

   • 13 UNIQUE constraint failed: session_message.session_id, session_message.seq errors in one serve log, across 4 sessions, 2026-10-05 → 2026-10-19.
   • The serve that failed was the only process holding the DB file and had been up since two days before the failing prompt.
   • Wedged session: 132 session_message rows, seq 2…1321 (sparse), last rows compaction@1318 and idle@1321 — event_sequence.seq for the session = 5. INSERT … seq=5 collides on the unique index; the session is suspended the same second the prompt arrives.
   • Fleet shape in the same DB: of the 16 sessions that have an event_sequence row, 14 have counter <= max(session_message.seq) — one reload away from the same wedge. The other 2151 sessions have no counter row at all (allocator derives the seq from the rows), which is why only sessions with a stale row die.

   So the counter is written around session creation and never advanced by the projection path; a session that lives inside one process is fine (in-memory counter), and the first reload after the row went stale is fatal. Repair that unblocked it (verified on a copy): event_sequence.seq = COALESCE(MAX(session_message.seq), -1) + 1 for the session, then restart.

   Full write-up with schema, numbers and the before/after repair transcript: #56975.
10. Seamus05 commented on Oct 26, 2026

    

    Seamus05

    on Oct 26, 2026

    More actions

    Follow-up: a live session's counter can sit thousands behind its own max(seq) (v2.1.4, Linux)

    Following up on our 2026-10-19 comment above, which established single-writer. This one is about where the counter goes wrong, because "stale counter" points at the writer and what we can see points at the read path.

    Observed

    Two long-lived sessions on this box (v2.1.4, Linux, single writer, one serve per store):

    session	event_sequence.seq	max(session_message.seq)
    A	45	7765
    B	13	2099

    Both were live and taking turns. In A, the first write after an ~5-hour idle window used seq 2. Turns then kept succeeding for about four hours — the allocator was walking A's own early range, where the slots were free. At seq 46 it reached a row from two days earlier; UNIQUE(session_message.session_id, seq) rejected the insert and the lifecycle publish died. Every turn after that failed. B was caught before collision by the check below.

    Repair was to set the counter to max(seq)+1 (7766); A re-verified healthy an hour later at 7934 vs max 7925.

    What this points at

    The row was present and readable — it held 45. The allocator did not use 45; the first post-idle write used 2. So the failure is in the read path after whatever happens on idle, not in the write. We read the allocator as falling back to -1 when the row cannot be read and then seeding from 0, but we have not reproduced that deliberately, so treat it as a hypothesis, not a finding.

    One thing that is not a symptom

    In this store only 5 of the 368 sessions that have messages also have an event_sequence row — and those 5 are exactly the sessions the serve is currently tracking. A missing row appears to be normal for a closed session, so it should not be used as a corruption signal. The signal is a live session whose seq sits behind its own max(session_message.seq).

    Detector

    Read-only, per store, cheap enough to run on a timer:

    select e.aggregate_id, e.seq, m.mx
    from event_sequence e
    join (select session_id, max(seq) as mx
          from session_message group by session_id) m
      on m.session_id = e.aggregate_id
    where e.seq < m.mx;

    Any row returned is a session already walking into a collision. Ours runs every 15 minutes across five stores; it repairs only when the session is idle, and records rather than repairs when it cannot read the serve. It currently reports 0 behind.

    Repro shape

    Idle a long-lived session past whatever rehydration threshold applies, then write once and watch the seq. We have not narrowed the threshold.
11. opencodejr commented on Oct 27, 2026

    

    opencodejrBot

    on Oct 27, 2026

    Contributor

    More actions

    Thanks for the detailed write-up and the DB evidence. I tried to reproduce this on 2.0.22 (Linux x64) and couldn't.

    What I tried: two opencode serve processes sharing one data directory (so one opencode.db), with a stub OpenAI-compatible model. I ran turns in the same session from both servers, first alternating and then concurrently (about 28 assistant turns). There was no UNIQUE constraint failed: session_message.session_id, session_message.seq and no "Failed to drain Session", and event_sequence.seq for the session stayed equal to max(session_message.seq).

    I don't think the cause is an in-memory counter. In v2, session_message.seq is the durable event's sequence number, which is read from the event_sequence table (latest + 1) in the same SQLite transaction that inserts the message row. So a second server reads the current value from the database. For a collision like yours, the event_sequence row for that session would have to fall behind the rows already in session_message.

    These would help us track it down:

    1. Versions: the output of opencode --version from both processes, the --service one and the one started by opencode -c. You mention the same setup exists on machines running 1.18.34: could a 1.x binary ever have opened the same opencode.db?
    2. Session origin: was the affected session created in v1 and continued in v2? That case is reported in core: continuing a V1-created session fails with UNIQUE constraint on event seq #52395.
    3. The counter row: for the affected session (on a copy of the DB), the result of select seq from event_sequence where aggregate_id = '<session id>', next to select max(seq) from session_message where session_id = '<session id>' and select max(seq) from event where aggregate_id = '<session id>'.
    4. Exact setup: the commands as you ran them, including whether both processes use the same XDG_DATA_HOME and channel.

    You can reply here or edit the issue. If you edit it, please also leave a comment saying you've updated it.

    Related:

    • core: continuing a V1-created session fails with UNIQUE constraint on event seq #52395: the same kind of UNIQUE failure on event.aggregate_id, event.seq after continuing a V1-created session, where the per-session sequence row was behind the stored rows.
12. opencodejr

    closed this as not plannedon Oct 30, 2026
13. opencodejr commented on Oct 30, 2026

    

    opencodejrBot

    on Oct 30, 2026

    Contributor

    More actions

    Closing this for now since we haven't heard back. We couldn't reproduce the UNIQUE constraint failed: session_message.session_id, session_message.seq error on 2.0.22 with two servers sharing one opencode.db and running turns in the same session. Without the details we asked for, we can't tell what makes the per-session event_sequence counter fall behind in your setup. Those details were: the versions of both processes (and whether a 1.x binary ever opened the same DB), whether the session was created in v1, and the event_sequence.seq vs max(seq) values for the affected session.

    If it happens again, please reply here with those details or open a new issue linking this one, and we'll pick it up. #52395 covers a related case: continuing a V1-created session. Thanks again for the thorough report.