[Bug]: OpenCode usage scan takes ~5 min on a large OpenCode 2 database, repeats on every refresh and stalls the server #28430

Activity

1. imadtg commented on Nov 7, 2026

   

   imadtg

   on Nov 7, 2026

   More actions

   Same reader, a much smaller database, and still minutes per summary: on a spinning disk the cost follows bytes read, not row count.

   Version: server 0.1.7-nightly.20261101.4016 (background service); current main (298cb24) has the same uncached query in packages/provider-opencode/src/server/usage.ts.
   Environment: Linux 7.2.8 (Omarchy), Node 24.22, OpenCode 2.1.4, 7.6 GB RAM, 5400 rpm SATA HDD under LUKS2 + Btrfs.

   Database: opencode.db is 453 MB but holds only about 8,000 message and 8,300 session_message rows (about 28 KB of data per row). The scan selects data for every in-window row, so with rows this large a summary can read most of the file from disk again.

   Server traces, 2026-11-04 21:41 to 2026-11-05 04:50. I excluded windows where other heavy disk jobs ran on this machine.

   	runs	median	max
   UsageService.scanSummary	7	62 s	123 s
   openCodeUsageReader.scan within it	7	61 s	121 s

   The OpenCode reader dominates even in scans where everything else is cheap. At 04:27 it took 87.8 s, while the Antigravity reader took 0.5 s and the transcript walk 0.3 s.

   Knock-on effect. In 3 of the 7 scans, the Antigravity reader's 9 per-database reads took 45–86 s in total. In the other 4 they took 0.4–7 s. The databases have not changed since October, so their fingerprints match and those reads should be cache hits: one realpath and two stat calls each. The extra time looks like queueing behind other I/O in the same scan, which is the shape described in #26417, but I have not confirmed that.

   What would help here, in order:

   1. Cache the OpenCode result like the transcript and Antigravity readers already do. Key it on the highest rowid seen and time_updated, so a repeat request only reads new or updated rows.
   2. Select only the fields the summary needs instead of the whole data blob. Where the schema allows it, extract them in SQL, so a row's context is not read and parsed for its token counts.
   3. Keep the scan off the request path and return partial totals meanwhile, in line with the expected behaviour above, so a slow disk does not hold the Usage page and the RPC open.

   Related: #27667 (the same reader with unbounded page bytes), #26417 (filesystem work exhausting the libuv pool).

   Written by

   Claude Code (Claude Opus 5.5, claude-opus-5-5), following the t3 triage playbook, from measurements on the machine described above.
2. nkoynov commented on Nov 11, 2026

   

   nkoynov

   on Nov 11, 2026

   ContributorAuthor

   More actions

   Still happens on 0.1.8-nightly.20261107.4340. #33612 (in 4446 and later) makes each scan cheaper but leaves it on the event loop.

   On 2026-11-11 one open of the Usage page in the desktop app started two summaries 8 s apart (two windows). Each ran its own readOpenCodeUsage over a 13.3 GB OpenCode 2 database: 2,151 s and 2,153 s, side by side. While they ran:

   • event-loop-stalls.ndjson showed utilization 1.0 in every sample (max delays 2.1–5.6 s), and RSS went from 0.6 to 1.1 GB.
   • 55 % of orchestrationV2.dispatch.withReceipt took over 1 s (p90 15 s, max 28.8 s), against 7 % (p90 0.7 s) outside the scans over the surrounding 2.5 days. Point queries on T3's own database waited 15–17 s for the connection.
   • The server's main thread was blocked on disk reads (D state) in 12 of 30 one-second samples. The OpenCode database is on a busy disk (about 2 ms per read).
   • The desktop app hit its 15 s ping timeout, reconnected through the relay, and moved back to the direct route two minutes later, re-subscribing everything each time.
   • The page's two requests were interrupted after about 10 minutes, but the scans run detached and kept going, so closing the page doesn't end the stall.

   The same happened on 2026-11-09 (two scans, 1,272 s and 1,278 s; 43 % of dispatches over 1 s). A second server with a 2 GB database ran 22 scans of up to 85 s each within six minutes on 2026-11-10.

   On 4515 the reader still opens NodeSqliteClient.layer({ filename, readonly: true }) without readerWorker, so every page is a synchronous node:sqlite call on the event loop. A 30-day window covers most of an active database (91 % of the 750k session_message rows here). Opening it with readerWorker: true, as T3's own database does, and piping sql.withTransaction(read) through NodeSqliteClient.readOnly would move the transaction and every statement in it to the worker (the transaction picks its connection when it starts, so the wrapper has to be outside it). The database is WAL, which the reader worker requires. If the worker can't start, the pool turns off and reads fall back to the event loop with only a warning in the log, so this reader may be better off refusing in that case.

   Workaround here: a preload that refuses session_message/message statements on read-only connections to opencode*.db. The reader then reports "Some OpenCode history could not be read." and the page shows the other sources.

   Model and harness: Claude Opus 5.5 in OpenCode, via T3 Code.
3. juliusmarminge commented on Nov 13, 2026

   

   juliusmarminge

   on Nov 13, 2026

   Member

   More actions

   This is the same problem as #34897, so I'm closing it in favour of that one. If it is different, say how and it will be reopened.
4. juliusmarminge

   closed this as duplicateon Nov 13, 2026
5. nkoynov commented on Nov 14, 2026

   

   nkoynov

   on Nov 14, 2026

   ContributorAuthor

   More actions

   Retested #34897 on nightly 4669 (db2cc20) with OpenCode 2.3.0. The stall part is fixed: during a 9.4 s cold scan of a 1.96 GB opencode.db, other HTTP requests answered in under 15 ms.

   The repeat part only partly is. The row cache's version is the database file's identity, and that includes the main file's size and mtime. Both change at every WAL checkpoint, so the first refresh after a write that reaches a checkpoint drops the whole cache and rereads every body. In the VM, a refresh with nothing changed took 0.9 s, and one after a single turn took 3.2 s. On a busy server that's most refreshes.

   Both tables here have time_updated and the per-row check already compares it with octet_length. So would it be enough to key the version on dev, inode and birth time only, to catch a replaced file, and let the per-row markers pick up new and changed rows?