Skip to content

Commit 363b134

Browse files
irparentclaude
andauthored
One copy of SQLite per database file in a process, and prove worker threads end cleanly (#739) (#733)
* One copy of SQLite per database file in a process (#739) better-sqlite3 carries its own SQLite and node:sqlite is Node's. Their POSIX locks belong to the process, so two copies on one WAL file cannot see each other: one resets the -shm file the other has mapped, and the other dies of SIGBUS (or the file is corrupted). A test's stand-in search thread did this, and the test process died of SIGBUS in 2 of 30 runs on macOS. On Linux, node:sqlite on the main thread with better-sqlite3 on a worker died in 3 runs of 5; one copy on both, 0 of 10. openDriver now keeps, per thread, which copy holds each open file: the other copy is refused before it opens anything, and an unset choice follows the file and never falls back. A test pins the search worker and the checkpoint worker to the store's copy on both drivers. The stand-in thread opens the store with the store's copy. A new CI job ends worker threads that hold a connection every way Iris ends them (close, throw, terminate idle, in a JS function, in a SQLite statement, in a loop; Iris's search and checkpoint workers from the sources and the build), on Linux, macOS and Windows, Node 22 and 24, both drivers, and requires every process to survive; shows two copies against one; and runs the search worker's test file 50 times with coverage. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * A second close of a connection does nothing; the driver tests follow where native is refused - The seam's connection count called the driver's close() before checking whether that connection was already closed. node:sqlite throws on a second close ("database is not open"); better-sqlite3 does not. The file is still let go only after a close that succeeded. A driver test now closes a built-in connection twice on every cell that has the built-in driver. - The job that builds the addon from source refuses native on Node 24 (nodejs/node#65446). There the worker test asserts that refusal and runs the drivers that open, and the connection-count test skips reopening the file with a copy that cannot load. - The two-copies CI step splits each pair explicitly and quotes both halves (shellcheck SC2086). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * search-worker tests get 30 s: under coverage on Windows two once passed 5 s The worker-exit job runs search-worker.test.ts 50 times with coverage on. On windows-latest with Node 22, run 11 of 50 had two tests pass vitest's 5 s default ("a delete during a search does not wait for it", "closes its connection with the store"). No thread or process died in any of the 50 runs or in the stress and two-copies steps before them. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Worker-exit fixture: reference a store's thread before terminating it, as the store's close does The checkpoint-kill case terminated the checkpoint thread directly. The Checkpointer unreferences an idle thread so it never holds a process open; a thread still in a native statement ends only when the statement returns, so with the thread unreferenced the event loop could empty first and Node exited 13 with the terminate() await unsettled (macOS, Node 24, 1 process in 10). Checkpointer.close() already references the thread before it waits for it; the fixture now does the same for its two direct terminations. Locally, 80 processes (src and dist, both drivers, both checkpoint endings) ran clean. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Checkpointer: hold the thread from close() on; the fixture holds the process while it terminates The reference the last commit added before terminate() was not enough: the Checkpointer unreferences its thread whenever an answer empties its queue, and the checkpoint-kill case's TRUNCATE answer arrived after it, so Windows with Node 22 still saw 1 process in 10 exit 13. - Product: the message handler and whenStarted() no longer unreference the thread once close() has begun (close sets failed, then holds the thread until it ends). Before, a checkpoint answer arriving during close() released it again; a thread that outlasted the close timeout could then leave a short-lived process exiting 13 under close(). A test spies on unref through a close with a TRUNCATE in flight: red before (called once, both drivers), green after. - Fixture: terminated() also holds the process with a timer while terminate() runs, as a server would, so a thread that never ends shows as a hang the stress harness counts. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * claims: recapture (#739) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent d4e36bb commit 363b134

12 files changed

Lines changed: 653 additions & 14 deletions

File tree

‎.claims.json‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5049,8 +5049,8 @@
50495049
],
50505050
"version": 1
50515051
},
5052-
"generatedAt": "2026-09-30T22:39:55.009Z",
5053-
"generatedFromCommit": "2be9a907",
5052+
"generatedAt": "2026-10-01T00:32:03.502Z",
5053+
"generatedFromCommit": "886460a5",
50545054
"generatorVersion": "1.0.0",
50555055
"llmJudgeTemplates": {
50565056
"count": 7,
@@ -9266,16 +9266,16 @@
92669266
"passed": null,
92679267
"total": 51
92689268
},
9269-
"totalCombined": 4558,
9269+
"totalCombined": 4563,
92709270
"vitestDashboard": {
92719271
"failed": 0,
92729272
"passed": 423,
92739273
"total": 423
92749274
},
92759275
"vitestRoot": {
92769276
"failed": 0,
9277-
"passed": 4084,
9278-
"total": 4084
9277+
"passed": 4089,
9278+
"total": 4089
92799279
}
92809280
},
92819281
"version": {

‎.github/workflows/ci.yml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,76 @@ jobs:
281281
grep -q "driver better-sqlite3" self-test.txt
282282
fi
283283
284+
# Iris runs database connections on worker threads (the search worker and
285+
# the checkpoint worker). test (22, macOS) once died of SIGBUS while
286+
# search-worker.test.ts started and ended search threads; this job found
287+
# why. Ending a worker is safe: a worker with its own connection, started
288+
# and ended thousands of times every way one ends in Iris (it closes its
289+
# own connection; it throws; terminate() while idle, in a JS function a
290+
# statement calls, in a statement in SQLite alone, in a loop of
291+
# statements), on both drivers, then Iris's own two workers every way each
292+
# ends, from the sources and from the build: no process died. What kills
293+
# the process is two copies of SQLite on one WAL file in one process
294+
# (better-sqlite3's and node:sqlite; driver.ts, "One copy of SQLite per
295+
# file"), which the test's stand-in thread did. The two-copies step shows
296+
# the mechanism on this OS and requires one copy to be clean; the last
297+
# step runs the search worker's test file in a loop with coverage on, as
298+
# the test job does, and requires every run to pass.
299+
worker-exit:
300+
name: worker exit (${{ matrix.os }}, Node ${{ matrix.node-version }})
301+
runs-on: ${{ matrix.os }}
302+
strategy:
303+
fail-fast: false
304+
matrix:
305+
os: [ubuntu-latest, macos-latest, windows-latest]
306+
node-version: [22, 24]
307+
defaults:
308+
run:
309+
shell: bash
310+
steps:
311+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
312+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
313+
with:
314+
node-version: ${{ matrix.node-version }}
315+
cache: npm
316+
cache-dependency-path: package-lock.json
317+
- run: npm ci
318+
- run: npm run build:server
319+
- name: Workers started and ended every way, on both drivers, raw and as Iris's two workers
320+
env:
321+
WORKER_EXIT_SEARCH: '1'
322+
run: node tests/fixtures/worker-exit/stress.cjs > worker-exit.json
323+
- name: Two copies of SQLite on one file in one process, against one copy
324+
if: always()
325+
run: |
326+
bad=0
327+
for pair in "native native" "node node" "native node" "node native"; do
328+
set -- "${pair% *}" "${pair#* }"
329+
died=0
330+
for i in $(seq 1 10); do
331+
node tests/fixtures/worker-exit/two-sqlites.cjs "$1" "$2" 100 > out.txt 2> err.txt || { died=$((died + 1)); echo " main=$1 worker=$2 run $i: exit $? $(grep -v -E 'ExperimentalWarning|trace-warnings' err.txt | head -1)"; }
332+
done
333+
echo "main=$1 worker=$2: $died of 10 processes died"
334+
# One copy on both threads must never die; two copies are shown, not required to die.
335+
if [ "$1" = "$2" ] && [ "$died" != 0 ]; then bad=1; fi
336+
done
337+
test "$bad" = 0
338+
- name: The search worker's test file, 50 times with coverage on
339+
if: always()
340+
run: |
341+
died=0
342+
for i in $(seq 1 50); do
343+
if ! npx vitest run tests/unit/storage/search-worker.test.ts --coverage.enabled --coverage.thresholds.lines=0 --coverage.thresholds.branches=0 --coverage.thresholds.functions=0 --coverage.thresholds.statements=0 > run.log 2>&1; then
344+
died=$((died + 1))
345+
echo "::error::run $i failed"
346+
grep -E "SIG[A-Z]+|exited unexpectedly|FAIL|Error" run.log | head -10
347+
fi
348+
done
349+
echo "search-worker.test.ts: $died of 50 runs failed"
350+
test "$died" = 0
351+
- if: always()
352+
run: cat worker-exit.json || true
353+
284354
integration:
285355
runs-on: ubuntu-latest
286356
needs: test

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,8 +174,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
174174
### Fixed
175175

176176
- **Closing the store releases the file before it returns (#750).** The checkpoint worker's connection was asked to close and given 5 s; past that the thread was terminated and `close()` returned at once, while the thread was still in its statement and still held `iris.db` open. On Windows the file could not then be moved or removed until that statement ended, and two storage suites failed on `windows-latest` removing their directories (`EPERM`) after every assertion had passed. `close()` now resolves only once the thread has ended, and answers any request the thread was in with an error rather than leaving it waiting. A test holds the worker in a four-second statement with a 100 ms timeout: before the change `close()` returned with the thread still running (4 runs in 4, both drivers); now it waits for it.
177+
- The checkpoint thread stays held from the start of `close()` until it has ended. The answer to a checkpoint in flight used to release it again. If the thread then outlasted `close()`'s timeout, a short-lived process (`iris-eval ingest`, say) could exit with code 13 while `close()` still waited. The worker-exit CI job showed that release when ending the thread directly, in 1 process in 10.
177178
- **A search's page of trace ids, and the start's check for evaluations with no stored risk estimate, keep their plans when the file has statistics (#742).** Iris never writes statistics, but `ANALYZE` run by hand does, and under them SQLite read the search index's whole trace_id index to find a page's ids, and every evaluation to find the ones without a risk estimate, at every start. The page's ids are now looked up one doc id at a time in a fixed join, in one statement for any number of matches (100,000 in 80 ms at 100,000 traces, against 92 ms in chunks of 1,000), and the fill names its index. The query-plan test covers every export read, and `getEvalsByTraceIds` names the index it already used.
178179
- **The website builds only when something its build reads has changed.** Every pull request used to create a preview build of iris-eval.com, and one day of pull requests used up the host's daily build allowance, which the live site's production deploys share. A preview now builds only for a change to `website/` or `docs/blog/`, and a production deploy also for a `.claims.json` change other than its timestamp and commit stamp. A branch's first deployment compares with `main`, and a base the host's shallow clone lacks is fetched; anything that fails builds. `tests/website-build-scope.test.ts` runs the real skip script in a shallow clone, as the host does, for six kinds of change, a branch's first deployment and an unreachable base, and fails if a website source reads any other path in the repository. CI's website job now runs the production build (`next build`) on every pull request, so a `.claims.json` change that would break the site is still caught without a preview.
180+
- **A database file is never opened by both copies of SQLite in one process (#739).** better-sqlite3 carries its own SQLite and `node:sqlite` is Node's. Their locks cannot see each other within one process, and one can reset the WAL's shared-memory file while the other has it mapped, which kills the process with SIGBUS or corrupts the file. Measured on Linux with Node 24.21.0: SIGBUS in 3 runs of 5, against 0 of 10 with one copy. Iris never did this: the store and its two worker threads use the copy the store's connection got. The one place it happened was a test's stand-in search thread, which died of SIGBUS in 2 of 30 runs on macOS. Now the driver seam enforces it: a file already open in a thread is opened again only with the copy that holds it, and the other copy is refused before it opens anything. A new CI job ends worker threads that hold a connection every way Iris ends them, on Linux, macOS and Windows, Node 22 and 24, both drivers: 9,000 per cell, and no process died.
179181
- **Scoring a trace no longer reads every evaluation in the store (#711).** Each evaluation with a cost reads its agent's recent history (the failure log), and so do `/api/v1/moments`, `/api/v1/failures`, the views and webhook moment events. There are two ways to read that history, and which one is fast depends on the data. 0.19.0 always read every evaluation of the store; the covering index added for search moved SQLite to walking every trace instead; with statistics it picked the slow way for a store where every trace is evaluated. The read now chooses between two queries that name their index and fix their join order, using counts of index entries. At 100,000 traces with every trace evaluated, `log_trace` with `evaluate: true` took 291 ms in 0.19.0 and takes 25.7 ms, `POST /api/v1/traces` with evaluate 291 ms and 24.9 ms, and `GET /api/v1/moments` 2.1 s and 0.8 s (medians of three rounds). The failure-log read itself went from 160 to 176 ms to 3 ms. On a store with few evaluations, where 0.19.0 was already fast, it stays between 0.1 and 3.5 ms. A test checks every result against the plain query on 12 seeded random stores per driver, through all three of the read's paths.
180182
- **Every hot read keeps its query plan, whatever SQLite's statistics say (#711).** Iris never runs `ANALYZE`. With statistics from a store of 100,000 evaluated traces, SQLite made the failure log 35 times slower and the 30-day summary 2.4 times slower. The reads that run on every evaluation or dashboard render (the failure log, case results, `get_traces` with and without `q`, the trace search join, the filter lists, the summary and the eval stats) now name their index and fix their join order. `tests/unit/storage/query-plans.test.ts` checks each statement's plan on both drivers, with and without FTS5, with no statistics and under three sets of them. The test fails if a statement reads a whole table, or if a join starts from the wrong side or reaches the other side through the wrong index.
181183
- **The dashboard's first render reads index entries, not trace rows (#711).** At 100,000 traces: `GET /api/v1/filters` 198 ms in 0.19.0, 1.4 ms now (one index seek per distinct agent and framework); `GET /api/v1/summary?hours=720` 284 ms and 62 ms; `GET /api/v1/eval-stats?period=30d` 172 ms and 33 ms. `get_traces` sorted by `cost_usd` or `latency_ms` sorts index entries and reads the 50 rows of the page: 187 ms and 27 ms.

‎docs/architecture.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,21 @@ Registered rules are appended to the built-in set for that eval type and include
334334
- **Foreign keys enabled** (`PRAGMA foreign_keys = ON`): Ensures span and eval_result integrity.
335335
- **Database path**: Default `~/.iris/iris.db`. Configurable via `--db-path`, `IRIS_DB_PATH`, or config file.
336336

337+
### One copy of SQLite per file
338+
339+
The store holds more than one connection to its file: the adapter's own, and one on each worker thread (the search worker, `src/storage/search-worker.ts`, and the checkpoint worker, `src/storage/checkpointer.ts`). All of them must be opened by the same copy of SQLite. better-sqlite3 carries its own SQLite, and `node:sqlite` is Node's, so they are two copies of the library in one process.
340+
341+
SQLite's locks on a file are POSIX advisory locks, and those belong to the process. When both copies open one WAL database in one process, neither sees the other's locks. Each can take itself for the file's only user and reset the `-shm` file the other has mapped, and the other's next read of it is a SIGBUS. Closing a descriptor in one copy also drops the locks the other holds, which is how a file gets corrupted ([How To Corrupt An SQLite Database File, "Multiple copies of SQLite linked into the same application"](https://www.sqlite.org/howtocorrupt.html#multiple_copies_of_sqlite_linked_into_the_same_application)). Measured on Linux with Node 24.21.0 (`tests/fixtures/worker-exit/two-sqlites.cjs`): with `node:sqlite` on the main thread and better-sqlite3 on a worker, the process died of SIGBUS in 3 runs of 5. With one copy on both threads, 0 of 10. Windows locks belong to the file handle, and there 0 of 20 died in any pairing.
342+
343+
The rule for every connection, and for every future worker thread:
344+
345+
- A worker is handed the driver the adapter's connection actually got (`Driver.name`), never the configured one. On a machine where the adapter fell back to `node:sqlite`, a worker told `native` would open the other copy.
346+
- A worker opens through `openDriver`, never `require('better-sqlite3')` or `node:sqlite` directly.
347+
- `openDriver` keeps, per thread, which copy holds each open file. It refuses the other copy before opening anything, and a connection with no driver chosen follows the copy that holds the file. It cannot see across threads, so `tests/unit/storage/driver.test.ts` pins both workers to the adapter's copy on both drivers.
348+
- A test's stand-in worker opens the store with `workerData.driver`, as the real one does.
349+
350+
Ending a worker is not a hazard. The `worker exit` CI job ends worker threads that hold a connection every way Iris ends them, on Linux, macOS and Windows, Node 22 and 24, both drivers: 9,000 per cell, and no process died. The endings are: the worker closes its own connection; it throws; `terminate()` while idle, inside a JS function a statement calls, inside a statement in SQLite alone, or inside a JS loop; and Iris's own two workers from the sources and from the build.
351+
337352
### Schema
338353

339354
Every data table carries a `tenant_id TEXT NOT NULL DEFAULT 'local'` column (added in migration 004, v0.4.0). OSS single-node deployments only ever see `'local'`; the column means shared or hosted storage could be added later without a painful data migration. No such deployment exists today. See §8 for the four-layer defense-in-depth enforcement.

‎src/storage/checkpointer.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,9 @@ export class Checkpointer {
136136
if (m.id !== undefined) {
137137
this.waiting.get(m.id)?.(m);
138138
this.waiting.delete(m.id);
139-
if (this.waiting.size === 0) this.worker.unref();
139+
// Not once close() has begun (it sets failed, then holds the thread until it ends): a reply that empties the queue
140+
// then would let a CLI's event loop empty while close() still waits for the thread, and Node exits 13 under it.
141+
if (this.waiting.size === 0 && !this.failed) this.worker.unref();
140142
}
141143
});
142144
this.worker.on('error', (err) => this.fail(err instanceof Error ? err.message : String(err)));
@@ -158,7 +160,7 @@ export class Checkpointer {
158160
try {
159161
return await this.started;
160162
} finally {
161-
if (this.waiting.size === 0) this.worker.unref();
163+
if (this.waiting.size === 0 && !this.failed) this.worker.unref();
162164
}
163165
}
164166

0 commit comments

Comments
 (0)