You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 363b134
Browse filesBrowse the repository at this point in the historyBrowse files
One copy of SQLite per database file in a process, and prove worker threads end cleanly (#739) (#733)
* One copy of SQLite per database file in a process (#739)
better-sqlite3 carries its own SQLite and node:sqlite is Node's. Their
POSIX locks belong to the process, so two copies on one WAL file cannot
see each other: one resets the -shm file the other has mapped, and the
other dies of SIGBUS (or the file is corrupted). A test's stand-in search
thread did this, and the test process died of SIGBUS in 2 of 30 runs on
macOS. On Linux, node:sqlite on the main thread with better-sqlite3 on a
worker died in 3 runs of 5; one copy on both, 0 of 10.
openDriver now keeps, per thread, which copy holds each open file: the
other copy is refused before it opens anything, and an unset choice
follows the file and never falls back. A test pins the search worker and
the checkpoint worker to the store's copy on both drivers. The stand-in
thread opens the store with the store's copy.
A new CI job ends worker threads that hold a connection every way Iris
ends them (close, throw, terminate idle, in a JS function, in a SQLite
statement, in a loop; Iris's search and checkpoint workers from the
sources and the build), on Linux, macOS and Windows, Node 22 and 24, both
drivers, and requires every process to survive; shows two copies against
one; and runs the search worker's test file 50 times with coverage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* A second close of a connection does nothing; the driver tests follow where native is refused
- The seam's connection count called the driver's close() before checking
whether that connection was already closed. node:sqlite throws on a second
close ("database is not open"); better-sqlite3 does not. The file is still
let go only after a close that succeeded. A driver test now closes a
built-in connection twice on every cell that has the built-in driver.
- The job that builds the addon from source refuses native on Node 24
(nodejs/node#65446). There the worker test asserts that refusal and runs the
drivers that open, and the connection-count test skips reopening the file
with a copy that cannot load.
- The two-copies CI step splits each pair explicitly and quotes both halves
(shellcheck SC2086).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* search-worker tests get 30 s: under coverage on Windows two once passed 5 s
The worker-exit job runs search-worker.test.ts 50 times with coverage on.
On windows-latest with Node 22, run 11 of 50 had two tests pass vitest's
5 s default ("a delete during a search does not wait for it", "closes its
connection with the store"). No thread or process died in any of the 50
runs or in the stress and two-copies steps before them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* Worker-exit fixture: reference a store's thread before terminating it, as the store's close does
The checkpoint-kill case terminated the checkpoint thread directly. The
Checkpointer unreferences an idle thread so it never holds a process open;
a thread still in a native statement ends only when the statement returns,
so with the thread unreferenced the event loop could empty first and Node
exited 13 with the terminate() await unsettled (macOS, Node 24, 1 process
in 10). Checkpointer.close() already references the thread before it waits
for it; the fixture now does the same for its two direct terminations.
Locally, 80 processes (src and dist, both drivers, both checkpoint endings)
ran clean.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* Checkpointer: hold the thread from close() on; the fixture holds the process while it terminates
The reference the last commit added before terminate() was not enough: the
Checkpointer unreferences its thread whenever an answer empties its queue,
and the checkpoint-kill case's TRUNCATE answer arrived after it, so Windows
with Node 22 still saw 1 process in 10 exit 13.
- Product: the message handler and whenStarted() no longer unreference the
thread once close() has begun (close sets failed, then holds the thread
until it ends). Before, a checkpoint answer arriving during close()
released it again; a thread that outlasted the close timeout could then
leave a short-lived process exiting 13 under close(). A test spies on
unref through a close with a TRUNCATE in flight: red before (called once,
both drivers), green after.
- Fixture: terminated() also holds the process with a timer while
terminate() runs, as a server would, so a thread that never ends shows as
a hang the stress harness counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* claims: recapture (#739)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
echo "main=$1 worker=$2: $died of 10 processes died"
334
+
# One copy on both threads must never die; two copies are shown, not required to die.
335
+
if [ "$1" = "$2" ] && [ "$died" != 0 ]; then bad=1; fi
336
+
done
337
+
test "$bad" = 0
338
+
- name: The search worker's test file, 50 times with coverage on
339
+
if: always()
340
+
run: |
341
+
died=0
342
+
for i in $(seq 1 50); do
343
+
if ! npx vitest run tests/unit/storage/search-worker.test.ts --coverage.enabled --coverage.thresholds.lines=0 --coverage.thresholds.branches=0 --coverage.thresholds.functions=0 --coverage.thresholds.statements=0 > run.log 2>&1; then
344
+
died=$((died + 1))
345
+
echo "::error::run $i failed"
346
+
grep -E "SIG[A-Z]+|exited unexpectedly|FAIL|Error" run.log | head -10
347
+
fi
348
+
done
349
+
echo "search-worker.test.ts: $died of 50 runs failed"
Copy file name to clipboardExpand all lines: CHANGELOG.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -174,8 +174,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
174
174
### Fixed
175
175
176
176
-**Closing the store releases the file before it returns (#750).** The checkpoint worker's connection was asked to close and given 5 s; past that the thread was terminated and `close()` returned at once, while the thread was still in its statement and still held `iris.db` open. On Windows the file could not then be moved or removed until that statement ended, and two storage suites failed on `windows-latest` removing their directories (`EPERM`) after every assertion had passed. `close()` now resolves only once the thread has ended, and answers any request the thread was in with an error rather than leaving it waiting. A test holds the worker in a four-second statement with a 100 ms timeout: before the change `close()` returned with the thread still running (4 runs in 4, both drivers); now it waits for it.
177
+
- The checkpoint thread stays held from the start of `close()` until it has ended. The answer to a checkpoint in flight used to release it again. If the thread then outlasted `close()`'s timeout, a short-lived process (`iris-eval ingest`, say) could exit with code 13 while `close()` still waited. The worker-exit CI job showed that release when ending the thread directly, in 1 process in 10.
177
178
-**A search's page of trace ids, and the start's check for evaluations with no stored risk estimate, keep their plans when the file has statistics (#742).** Iris never writes statistics, but `ANALYZE` run by hand does, and under them SQLite read the search index's whole trace_id index to find a page's ids, and every evaluation to find the ones without a risk estimate, at every start. The page's ids are now looked up one doc id at a time in a fixed join, in one statement for any number of matches (100,000 in 80 ms at 100,000 traces, against 92 ms in chunks of 1,000), and the fill names its index. The query-plan test covers every export read, and `getEvalsByTraceIds` names the index it already used.
178
179
-**The website builds only when something its build reads has changed.** Every pull request used to create a preview build of iris-eval.com, and one day of pull requests used up the host's daily build allowance, which the live site's production deploys share. A preview now builds only for a change to `website/` or `docs/blog/`, and a production deploy also for a `.claims.json` change other than its timestamp and commit stamp. A branch's first deployment compares with `main`, and a base the host's shallow clone lacks is fetched; anything that fails builds. `tests/website-build-scope.test.ts` runs the real skip script in a shallow clone, as the host does, for six kinds of change, a branch's first deployment and an unreachable base, and fails if a website source reads any other path in the repository. CI's website job now runs the production build (`next build`) on every pull request, so a `.claims.json` change that would break the site is still caught without a preview.
180
+
-**A database file is never opened by both copies of SQLite in one process (#739).** better-sqlite3 carries its own SQLite and `node:sqlite` is Node's. Their locks cannot see each other within one process, and one can reset the WAL's shared-memory file while the other has it mapped, which kills the process with SIGBUS or corrupts the file. Measured on Linux with Node 24.21.0: SIGBUS in 3 runs of 5, against 0 of 10 with one copy. Iris never did this: the store and its two worker threads use the copy the store's connection got. The one place it happened was a test's stand-in search thread, which died of SIGBUS in 2 of 30 runs on macOS. Now the driver seam enforces it: a file already open in a thread is opened again only with the copy that holds it, and the other copy is refused before it opens anything. A new CI job ends worker threads that hold a connection every way Iris ends them, on Linux, macOS and Windows, Node 22 and 24, both drivers: 9,000 per cell, and no process died.
179
181
- **Scoring a trace no longer reads every evaluation in the store (#711).** Each evaluation with a cost reads its agent's recent history (the failure log), and so do `/api/v1/moments`, `/api/v1/failures`, the views and webhook moment events. There are two ways to read that history, and which one is fast depends on the data. 0.19.0 always read every evaluation of the store; the covering index added for search moved SQLite to walking every trace instead; with statistics it picked the slow way for a store where every trace is evaluated. The read now chooses between two queries that name their index and fix their join order, using counts of index entries. At 100,000 traces with every trace evaluated, `log_trace` with `evaluate: true` took 291 ms in 0.19.0 and takes 25.7 ms, `POST /api/v1/traces` with evaluate 291 ms and 24.9 ms, and `GET /api/v1/moments` 2.1 s and 0.8 s (medians of three rounds). The failure-log read itself went from 160 to 176 ms to 3 ms. On a store with few evaluations, where 0.19.0 was already fast, it stays between 0.1 and 3.5 ms. A test checks every result against the plain query on 12 seeded random stores per driver, through all three of the read's paths.
180
182
-**Every hot read keeps its query plan, whatever SQLite's statistics say (#711).** Iris never runs `ANALYZE`. With statistics from a store of 100,000 evaluated traces, SQLite made the failure log 35 times slower and the 30-day summary 2.4 times slower. The reads that run on every evaluation or dashboard render (the failure log, case results, `get_traces` with and without `q`, the trace search join, the filter lists, the summary and the eval stats) now name their index and fix their join order. `tests/unit/storage/query-plans.test.ts` checks each statement's plan on both drivers, with and without FTS5, with no statistics and under three sets of them. The test fails if a statement reads a whole table, or if a join starts from the wrong side or reaches the other side through the wrong index.
181
183
-**The dashboard's first render reads index entries, not trace rows (#711).** At 100,000 traces: `GET /api/v1/filters` 198 ms in 0.19.0, 1.4 ms now (one index seek per distinct agent and framework); `GET /api/v1/summary?hours=720` 284 ms and 62 ms; `GET /api/v1/eval-stats?period=30d` 172 ms and 33 ms. `get_traces` sorted by `cost_usd` or `latency_ms` sorts index entries and reads the 50 rows of the page: 187 ms and 27 ms.
-**Database path**: Default `~/.iris/iris.db`. Configurable via `--db-path`, `IRIS_DB_PATH`, or config file.
336
336
337
+
### One copy of SQLite per file
338
+
339
+
The store holds more than one connection to its file: the adapter's own, and one on each worker thread (the search worker, `src/storage/search-worker.ts`, and the checkpoint worker, `src/storage/checkpointer.ts`). All of them must be opened by the same copy of SQLite. better-sqlite3 carries its own SQLite, and `node:sqlite` is Node's, so they are two copies of the library in one process.
340
+
341
+
SQLite's locks on a file are POSIX advisory locks, and those belong to the process. When both copies open one WAL database in one process, neither sees the other's locks. Each can take itself for the file's only user and reset the `-shm` file the other has mapped, and the other's next read of it is a SIGBUS. Closing a descriptor in one copy also drops the locks the other holds, which is how a file gets corrupted ([How To Corrupt An SQLite Database File, "Multiple copies of SQLite linked into the same application"](https://www.sqlite.org/howtocorrupt.html#multiple_copies_of_sqlite_linked_into_the_same_application)). Measured on Linux with Node 24.21.0 (`tests/fixtures/worker-exit/two-sqlites.cjs`): with `node:sqlite` on the main thread and better-sqlite3 on a worker, the process died of SIGBUS in 3 runs of 5. With one copy on both threads, 0 of 10. Windows locks belong to the file handle, and there 0 of 20 died in any pairing.
342
+
343
+
The rule for every connection, and for every future worker thread:
344
+
345
+
- A worker is handed the driver the adapter's connection actually got (`Driver.name`), never the configured one. On a machine where the adapter fell back to `node:sqlite`, a worker told `native` would open the other copy.
346
+
- A worker opens through `openDriver`, never `require('better-sqlite3')` or `node:sqlite` directly.
347
+
-`openDriver` keeps, per thread, which copy holds each open file. It refuses the other copy before opening anything, and a connection with no driver chosen follows the copy that holds the file. It cannot see across threads, so `tests/unit/storage/driver.test.ts` pins both workers to the adapter's copy on both drivers.
348
+
- A test's stand-in worker opens the store with `workerData.driver`, as the real one does.
349
+
350
+
Ending a worker is not a hazard. The `worker exit` CI job ends worker threads that hold a connection every way Iris ends them, on Linux, macOS and Windows, Node 22 and 24, both drivers: 9,000 per cell, and no process died. The endings are: the worker closes its own connection; it throws; `terminate()` while idle, inside a JS function a statement calls, inside a statement in SQLite alone, or inside a JS loop; and Iris's own two workers from the sources and from the build.
351
+
337
352
### Schema
338
353
339
354
Every data table carries a `tenant_id TEXT NOT NULL DEFAULT 'local'` column (added in migration 004, v0.4.0). OSS single-node deployments only ever see `'local'`; the column means shared or hosted storage could be added later without a painful data migration. No such deployment exists today. See §8 for the four-layer defense-in-depth enforcement.
0 commit comments