Skip to content

fix(context): keep context reads working when the wal-index cannot be established - #5432

Merged
thedotmack merged 4 commits into
thedotmack:mainfrom
0xABCD01:fix/5429-context-read-wal
Nov 6, 2026
Merged

thedotmack merged 4 commits into
thedotmack:mainfrom
0xABCD01:fix/5429-context-read-wal

Conversation

@0xABCD01

@0xABCD01 0xABCD01 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #5429

Since 15.9.0, every read the worker sends through its context-read worker thread fails with unable to open database file: context injection, date-sorted search, and context-cache re-renders, while main-thread reads and writes against the same database keep working.

The reader opens the database read-only (context-read-worker.ts, and ContextBuilder for renders). A read-only open of a WAL database needs the wal-index (-shm file). When the index is missing and the process cannot establish it for a read-only handle, the open does not fail at construction — it fails at the first statement, for every read, for the life of the process. In the reporter's tests the standalone reader works; the in-service failure is consistent with the wal-index being unestablishable for a read-only handle inside that process, and the test below constructs exactly that state.

What changed in 15.9.0 is not the reader: the read-only open code is identical between v15.8.3 and v15.9.0, and I could not pin the in-process trigger — why the worker service process cannot establish the wal-index for a read-only handle while the same reader code standalone can — to a single commit.

What this change does is remove the reader's hard dependency on that precondition: when a read-only open cannot be probed, a short-lived read-write open establishes the wal-index, and failing that the reader reads a snapshot copy. It does not identify or fix whatever changed on the 15.9.0 side to make the precondition start failing.

This adds openReaderDatabase in services/sqlite/connection.ts and uses it at the three read-only open sites. The read-only open is unchanged on the healthy path, except it is probed (PRAGMA schema_version via run(), after the busy timeout) so the lazy failure surfaces at the open. Only when the probe fails:

  1. A short-lived read-write open establishes the wal-index and the read-only open is retried while it is held — the handle callers read from is still read-only.
  2. If the index cannot be established in place at all (for example the data directory is not writable by this process), the reader reads a snapshot copy instead: the -wal is copied alongside the database, so opening the copy replays its committed frames, and the copy is deleted when the handle closes.

If every step fails (a missing or corrupt database), the original read-only error is rethrown, and nothing is created at the database path. readDatabase still applies query_only, and keyword search still gets a readOnly SessionSearch.

Tests: tests/context/context-read-wal-index.test.ts drives the real ContextReadWorkerClient against a database whose last session exists only in the -wal, with the -shm removed and wal-index establishment blocked in place. On main, every read returns the reported error; with this change the reads succeed and find the uncheckpointed project. The healthy-database and missing-database cases are pinned as unchanged. The full tests/context and tests/sqlite suites and npm run typecheck pass.

… established

The context read worker and context generation open the WAL database
read-only. That open needs the wal-index (-shm): when it is missing and
the process cannot establish it for a read-only handle, every read
through the worker fails with "unable to open database file" while the
writer in the same process keeps working (thedotmack#5429).

openReaderDatabase (services/sqlite/connection.ts) probes the read-only
open so the lazy failure surfaces at the open, not at the caller's first
query, and falls back only when it has failed: first a short-lived
read-write open establishes the wal-index and the read-only open is
retried under it; if the index cannot be established in place at all,
the reader reads a snapshot copy (the -wal is copied alongside, so its
committed frames replay) that close() deletes. Healthy and missing
databases behave exactly as before.

Tests: tests/context/context-read-wal-index.test.ts drives the real
ContextReadWorkerClient against a database whose frames are still only
in the -wal, with the -shm removed and the data directory made
unwritable for the reader. It fails on main with the reported error and
passes with this change; the healthy and missing-database cases are
pinned as unchanged.
@greptile-apps

greptile-apps Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest change introduces no new blocking issue.

Summary

Adds openReaderDatabase and uses it for context reads.

  • Local context reads recover when SQLite cannot set up the WAL index.

Reviews (4) · Last reviewed commit: "test(context): require the read-only fai..." · Reviewed by Greptile

Comment thread src/services/sqlite/connection.ts Resolved
… state

Review follow-up on thedotmack#5432 (thedotmack#5429).

The snapshot fallback copied the live database and its -wal to a temp
dir on every failed open. readDatabase opens per request, so an affected
1 GB database would be copied on every read (about 20,000 a day in the
report), the two copies could disagree (Greptile P1), and the keyword
search reader would cache a frozen copy for the thread's life (P1).
When a read-write open cannot create the -wal and wal-index either, the
original read-only error is now rethrown and both failures are logged.

The test now drives the reporter's own reproduction in a writable data
directory: a WAL database with neither -wal nor -shm. On main every
read returns "unable to open database file"; with this change reads and
date-sorted search succeed. A second case keeps an observation only in
the -wal with the -shm removed and asserts the reader still returns it
(through hasObservations and search, not readKeys, which echoes its
input; Greptile P2). The read-only mount/chmod setup is gone, which
fixes the CI EACCES failure and root-container runs (P2).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
thedotmack and others added 2 commits November 5, 2026 21:18
Review follow-up on thedotmack#5432: the no-wal/no-shm case now asserts that a
plain read-only open of the fixture fails with "unable to open database
file" before reading through the worker, so the read-write recovery is
always exercised rather than passing on a path that never needed it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
CI showed that the SQLite Bun embeds on Linux reads a WAL database with
no -wal and no -shm through a plain read-only open; the macOS system
library (the reporter's) refuses it. Require the precondition on macOS,
where it was observed, and keep the read assertions everywhere.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
@thedotmack
thedotmack merged commit 4f88958 into thedotmack:main Nov 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

15.9.0+: context-read worker fails every context render and date-sorted search with "unable to open database file" (15.8.3 works on the same DB)

2 participants