Repository navigation
fix(context): keep context reads working when the wal-index cannot be established - #5432
Merged
Merged
Conversation
… established The context read worker and context generation open the WAL database read-only. That open needs the wal-index (-shm): when it is missing and the process cannot establish it for a read-only handle, every read through the worker fails with "unable to open database file" while the writer in the same process keeps working (thedotmack#5429). openReaderDatabase (services/sqlite/connection.ts) probes the read-only open so the lazy failure surfaces at the open, not at the caller's first query, and falls back only when it has failed: first a short-lived read-write open establishes the wal-index and the read-only open is retried under it; if the index cannot be established in place at all, the reader reads a snapshot copy (the -wal is copied alongside, so its committed frames replay) that close() deletes. Healthy and missing databases behave exactly as before. Tests: tests/context/context-read-wal-index.test.ts drives the real ContextReadWorkerClient against a database whose frames are still only in the -wal, with the -shm removed and the data directory made unwritable for the reader. It fails on main with the reported error and passes with this change; the healthy and missing-database cases are pinned as unchanged.
Contributor
[Medium impact] The latest change introduces no new blocking issue. SummaryAdds
Reviews (4) · Last reviewed commit: "test(context): require the read-only fai..." · Reviewed by Greptile |
… state Review follow-up on thedotmack#5432 (thedotmack#5429). The snapshot fallback copied the live database and its -wal to a temp dir on every failed open. readDatabase opens per request, so an affected 1 GB database would be copied on every read (about 20,000 a day in the report), the two copies could disagree (Greptile P1), and the keyword search reader would cache a frozen copy for the thread's life (P1). When a read-write open cannot create the -wal and wal-index either, the original read-only error is now rethrown and both failures are logged. The test now drives the reporter's own reproduction in a writable data directory: a WAL database with neither -wal nor -shm. On main every read returns "unable to open database file"; with this change reads and date-sorted search succeed. A second case keeps an observation only in the -wal with the -shm removed and asserts the reader still returns it (through hasObservations and search, not readKeys, which echoes its input; Greptile P2). The read-only mount/chmod setup is gone, which fixes the CI EACCES failure and root-container runs (P2). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
Review follow-up on thedotmack#5432: the no-wal/no-shm case now asserts that a plain read-only open of the fixture fails with "unable to open database file" before reading through the worker, so the read-write recovery is always exercised rather than passing on a path that never needed it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
CI showed that the SQLite Bun embeds on Linux reads a WAL database with no -wal and no -shm through a plain read-only open; the macOS system library (the reporter's) refuses it. Require the precondition on macOS, where it was observed, and keep the read assertions everywhere. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5429
Since 15.9.0, every read the worker sends through its context-read worker thread fails with
unable to open database file: context injection, date-sorted search, and context-cache re-renders, while main-thread reads and writes against the same database keep working.The reader opens the database read-only (
context-read-worker.ts, andContextBuilderfor renders). A read-only open of a WAL database needs the wal-index (-shm file). When the index is missing and the process cannot establish it for a read-only handle, the open does not fail at construction — it fails at the first statement, for every read, for the life of the process. In the reporter's tests the standalone reader works; the in-service failure is consistent with the wal-index being unestablishable for a read-only handle inside that process, and the test below constructs exactly that state.What changed in 15.9.0 is not the reader: the read-only open code is identical between v15.8.3 and v15.9.0, and I could not pin the in-process trigger — why the worker service process cannot establish the wal-index for a read-only handle while the same reader code standalone can — to a single commit.
What this change does is remove the reader's hard dependency on that precondition: when a read-only open cannot be probed, a short-lived read-write open establishes the wal-index, and failing that the reader reads a snapshot copy. It does not identify or fix whatever changed on the 15.9.0 side to make the precondition start failing.
This adds
openReaderDatabaseinservices/sqlite/connection.tsand uses it at the three read-only open sites. The read-only open is unchanged on the healthy path, except it is probed (PRAGMA schema_versionviarun(), after the busy timeout) so the lazy failure surfaces at the open. Only when the probe fails:If every step fails (a missing or corrupt database), the original read-only error is rethrown, and nothing is created at the database path.
readDatabasestill appliesquery_only, and keyword search still gets areadOnlySessionSearch.Tests:
tests/context/context-read-wal-index.test.tsdrives the realContextReadWorkerClientagainst a database whose last session exists only in the -wal, with the -shm removed and wal-index establishment blocked in place. On main, every read returns the reported error; with this change the reads succeed and find the uncheckpointed project. The healthy-database and missing-database cases are pinned as unchanged. The fulltests/contextandtests/sqlitesuites andnpm run typecheckpass.