Skip to content

Commit b8449b3

Browse files
committed
fix(tests): disable V8 Sparkplug for jest to stop the worker SIGSEGV flake (bug 83)
The intermittent 'jest worker terminated by SIGSEGV' failure (~1 full run in 5, always green on rerun) was never the native SQLCipher binding: the macOS crash report shows the dying worker with no better_sqlite3.node loaded, and a faulting stack matching nodejs/node#62393 — a V8 13.6 GC race in ClearStaleLeftTrimmedPointerVisitor during a mark-compact fired inside Sparkplug's baseline prologue. Still reproduces upstream on Node 26, so the proven workaround is applied at two chokepoints: the jest npm scripts launch node --no-sparkplug directly (NODE_OPTIONS rejects the flag), and jest.global-setup.js appends it to process.execArgv before any worker forks so ad-hoc npx jest runs are covered too. The integration config now shares that globalSetup, gaining the native-ABI self-heal. Also: quantize-embeddings.test.ts now closes its per-test database. Verified: 9 consecutive full unit runs (half cold-cache) with zero new crash reports; integration, tsc, and lint clean.
1 parent e22f7b3 commit b8449b3

11 files changed

Lines changed: 176 additions & 24 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ No subscriptions. No data harvested. No forgetting between sessions. No landlord
88

99
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
1010
[![Latest Stable](https://img.shields.io/github/v/release/foundry-9/quilltap-server?logo=github&label=stable&sort=semver&filter=!*dev*)](https://github.com/foundry-9/quilltap-server/releases/latest)
11-
[![This Version](https://img.shields.io/badge/version-4.9.0--dev.37-yellow.svg?logo=github)](package.json)
11+
[![This Version](https://img.shields.io/badge/version-4.9.0--dev.38-yellow.svg?logo=github)](package.json)
1212
[![Docker Hub](https://img.shields.io/docker/v/foundry9/quilltap?logo=docker&label=docker&sort=semver)](https://hub.docker.com/r/foundry9/quilltap)
1313
[![npm](https://img.shields.io/npm/v/quilltap?logo=npm)](https://www.npmjs.com/package/quilltap)
1414
[![Discord](https://img.shields.io/badge/Discord-join-5865F2?logo=discord&logoColor=white)](https://discord.gg/6enCeQxY)

‎__tests__/unit/lib/database/migration/quantize-embeddings.test.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,12 @@ beforeEach(() => {
108108
);
109109
});
110110

111+
afterEach(() => {
112+
// Native handles left open across tests are the kind of loose end the other
113+
// real-binding suites all tie off — close per test, not at process exit.
114+
try { db.close(); } catch { /* already closed */ }
115+
});
116+
111117
describe('quantize-embeddings-v1', () => {
112118
it('declares its metadata and dependencies', () => {
113119
expect(quantizeEmbeddingsMigration.id).toBe('quantize-embeddings-v1');

‎docs/CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44

55
### 4.9-dev
66

7+
#### The intermittent jest worker segfault was V8's, not SQLCipher's (bug 83)
8+
9+
For months, roughly one full `npm run test:unit` run in five ended with `A jest worker process ... was terminated by another process: signal=SIGSEGV` failing one arbitrary suite while every actual test passed, and a rerun always went green. It was assumed to be the known native-SQLCipher teardown flake fixed in June. A macOS crash report proved otherwise: the dying worker had no `better_sqlite3.node` loaded at all, and the faulting stack matched [nodejs/node#62393](https://github.com/nodejs/node/issues/62393) frame-for-frame — a V8 13.6 (Node 24) GC race where a mark-compact triggered inside Sparkplug's baseline prologue dereferences a junk frame slot. Upstream still reproduces it on Node 26, so the fix is the thread's proven workaround: disable the Sparkplug baseline compiler for test runs. The five jest scripts in `package.json` now launch `node --no-sparkplug node_modules/jest/bin/jest.js` (the flag isn't allowed in `NODE_OPTIONS`), and `jest.global-setup.js` appends `--no-sparkplug` to `process.execArgv` before any worker forks, so ad-hoc `npx jest` runs get protected workers too — jest-worker inherits the parent's `execArgv`. The integration config now shares that globalSetup, which also gives it the native-ABI self-heal. Nine consecutive full runs (half with a cleared transform cache) produced zero crashes; upstream's matrices report the same at larger sample sizes with no measurable wall-time cost. Also tightened while in there: `quantize-embeddings.test.ts` now closes its per-test in-memory database like every other real-binding suite. Filed as [bug 83](developer/bugs/fixed/bug-83-v8-sparkplug-worker-segfault.md).
10+
711
#### Group scenes stop turning into committee meetings
812

913
In a multi-character chat, especially on weaker models, every character's turn converged on the same shape: open with a roll-call recap of what the previous speakers said, endorse all of it, claim "the one thing nobody has named yet," repeat the cast's coined phrases verbatim, and close by restating the group's action list. One observed chat had three characters in a row ending consecutive turns with the identical sentence. Three changes attack the loop:

‎docs/developer/bugs.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
11
# Bugs — defects surfaced by the v5 port
22

3-
**Last Updated**: 2026-08-19
3+
**Last Updated**: 2026-08-20
44
**Codebase**: Quilltap v4.9.0-dev
55
**Provenance**: the quilltap-v5 native port's differential harness, its
66
dogfood walks against a copy of real data, and — from Bug 62 — v4's own
77
feature-spec work and browser verification
8-
**Status**: Bugs **1–82** are **fixed in v4** — nothing in this catalogue is
9-
open. Bugs 81 and 82 (filed and fixed 2026-08-19) both come of a request being
8+
**Status**: Bugs **1–83** are **fixed in v4** — nothing in this catalogue is
9+
open. Bug 83 (filed and fixed 2026-08-20) is the intermittent jest worker
10+
SIGSEGV, misattributed for months to the native SQLCipher binding and finally
11+
traced by a macOS crash report to an upstream V8 GC race
12+
([nodejs/node#62393](https://github.com/nodejs/node/issues/62393)); the suite
13+
now runs with Sparkplug disabled at two chokepoints, and the v5 side owes
14+
nothing — Rust doesn't carry V8. Bugs 81 and 82 (filed and fixed 2026-08-19) both come of a request being
1015
shaped by one answer where two were needed. In **81** the answer is a boolean:
1116
`requiresApiKey` was asked both "must this provider have a key?" and "may it?",
1217
and for OpenAI-Compatible — the one provider spanning an unauthenticated
@@ -437,6 +442,7 @@ One row per bug, newest last. **Bug** links to the entry; **Fix site** and
437442
| 80 | [a project's story background is ignored inside the workspace](bugs/fixed/bug-80-project-background-ignored-in-workspace.md) | 2026-08-18 | 2026-08-18 | Medium | The workspace replaced the per-view `::before` background layer with one arbitrated backdrop that views must *report* to, and suppressed the old layer inside `.qt-workspace` — but `ProjectDetailView` was never converted, so its `--story-background-url` reaches the screen by neither route. What paints instead is `ProsperoView`'s subsystem background, still registered under the tab's id after the view drilled into a project | `ProjectDetailView` reports its story background (falling back to the Prospero subsystem image for `theme` mode); the list's subsystem background moved into a `ProsperoListShell` that unmounts while a detail is shown, so one reporter holds the tab's key at a time | Not applicable — the workspace shell and its backdrop arbitration have no v5 counterpart yet |
438443
| 81 | [an OpenAI-Compatible profile can never hold an API key](bugs/fixed/bug-81-oac-cannot-hold-an-api-key.md) | 2026-08-19 | 2026-08-19 | Medium | `requiresApiKey` answers two questions with one boolean, so OpenAI-Compatible is absent from the Add-New-API-Key provider list **and** from the profile form's key field — every hosted OpenAI-compatible endpoint that needs a bearer token is unconfigurable. Server-side, four call sites gated the key *lookup* on the same flag, so even an attached key never reached the wire | An optional `acceptsApiKey` capability answers the second question (`@quilltap/plugin-types` 2.5.7, absent = same answer as `requiresApiKey`); both UI gates and `useProfileForm`'s outbound guard read it via `lib/llm/api-key-support.ts`; the four services share `resolveConnectionProfileApiKey` | Owed |
439444
| 82 | [three leading system messages break strict local chat templates](bugs/fixed/bug-82-three-leading-system-messages.md) | 2026-08-19 | 2026-08-19 | High (for local models) | Every non-opening turn dies with `Jinja Exception: System message must be at the beginning` on Qwen-family templates — the greeting sends one system block and works, a normal turn sends three and is refused before a token is generated | `collapseLeadingSystemMessages` (`@quilltap/plugin-utils` 2.4.0) folds the leading run at request-build time; Ollama calls it unconditionally, OAC via `acceptsRepeatedSystemMessages: false` — the flag defaults true, so hosted subclasses stay byte-identical | Owed |
445+
| 83 | [a V8 GC race kills a jest worker and fails an arbitrary suite](bugs/fixed/bug-83-v8-sparkplug-worker-segfault.md) | 2026-08-20 | 2026-08-20 | Medium (dev tooling) | ~1 full unit run in 5 loses a worker to a SIGSEGV in V8's mark-compact ([nodejs/node#62393](https://github.com/nodejs/node/issues/62393)) and fails a different innocent suite each time; months of misattribution to the native SQLCipher binding trained a "just rerun it" reflex | `package.json` jest scripts (`node --no-sparkplug`) + `armSparkplugGuard()` in `jest.global-setup.js`, now shared by `jest.integration.config.ts` | Nothing owed (no V8 in Rust) |
440446

441447
### Families and reading order
442448

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Bug 83 — a V8 GC race kills a jest worker and fails an arbitrary suite
2+
3+
| | |
4+
|---|---|
5+
| **Status** | **Fixed in v4** (worked around; the defect itself is upstream V8's) |
6+
| **Found** | 2026-08-20 (endemic long before that — the crash signature appears in session logs at least back to 2026-08-11, misattributed the whole time) |
7+
| **Fixed** | 2026-08-20 |
8+
| **Severity** | Medium — no user data at risk, but roughly **1 full `npm run test:unit` run in 5** on the macOS arm64 dev machine lost a worker and failed one suite. Worse than the wasted rerun is the trained reflex: "red suite? run it again" is exactly the habit that lets a *real* intermittent failure slide by unexamined |
9+
| **Who it bites** | anyone running the jest suites on Node 24+ (the current engines floor). Reproduced upstream on macOS arm64; the defect is in shared V8 code, so other platforms are presumed exposed |
10+
| **Provenance** | Not this codebase's at all. V8 13.6 (Node 24) has a GC race — [nodejs/node#62393](https://github.com/nodejs/node/issues/62393), Chromium issue 483731079. Still reproduces upstream on Node 26.7.0, so waiting out a Node upgrade was not an option |
11+
| **Fix site** | `package.json` (the five jest scripts now launch `node --no-sparkplug node_modules/jest/bin/jest.js`), `jest.global-setup.js` (`armSparkplugGuard()` appends `--no-sparkplug` to `process.execArgv` before any worker forks), `jest.integration.config.ts` (now wired to the same globalSetup) |
12+
| **Defect site** | upstream: `v8::internal::ClearStaleLeftTrimmedPointerVisitor::VisitRootPointers` dereferencing a junk slot of an `InternalFrame` mid-`Builtins_BaselineOutOfLinePrologue` |
13+
| **v5 status** | Nothing owed — the Rust core doesn't run V8. The workaround should ride along only as long as v4's jest suites do |
14+
| **Index** | [bugs.md](../../bugs.md) |
15+
16+
---
17+
18+
**FIXED in v4 (2026-08-20)** — by disabling V8's Sparkplug baseline compiler for
19+
every jest process, at two chokepoints so no invocation style escapes:
20+
21+
1. The five jest scripts in `package.json` (`test:all`, `test:coverage`,
22+
`test:integration`, `test:unit`, `test:watch`) launch jest as
23+
`node --no-sparkplug node_modules/jest/bin/jest.js`. (`NODE_OPTIONS` can't
24+
carry the flag — it's not on Node's allowlist — hence the explicit
25+
interpreter invocation.) jest-worker forks each worker with the parent's
26+
`process.execArgv`, so the flag propagates to the whole fleet, and the main
27+
process — which runs the tests itself under `--runInBand` or a single-file
28+
run — is covered directly.
29+
2. `jest.global-setup.js` (`armSparkplugGuard()`) appends `--no-sparkplug` to
30+
`process.execArgv` when absent. `process.execArgv` is a plain mutable array
31+
that jest-worker reads at fork time, and globalSetup runs in the main
32+
process before any worker forks — so ad-hoc invocations that skip the npm
33+
scripts (`npx jest`, `npx jest -u some.test.ts`) still get protected
34+
workers. The integration config now points at the same globalSetup, which
35+
also brings it the native-ABI self-heal it never had.
36+
37+
## Symptom
38+
39+
Intermittently — about 1 full unit run in 5 locally — a run ends:
40+
41+
```
42+
FAIL __tests__/unit/<some suite>.test.ts
43+
● Test suite failed to run
44+
A jest worker process (pid=NNNNN) was terminated by another process:
45+
signal=SIGSEGV, exitCode=null.
46+
```
47+
48+
with **every individual test in the run passing**. A *different* suite fails
49+
each time — plugin tests, component tests, API tests — because the dying worker
50+
takes down whatever file it happens to hold. A rerun essentially always passes,
51+
which is what kept this filed under "flake" instead of "bug" for months.
52+
53+
## Root cause
54+
55+
A V8 13.6 garbage-collection race, [nodejs/node#62393](https://github.com/nodejs/node/issues/62393).
56+
When a stack-guard interrupt fires inside Sparkplug's
57+
`Builtins_BaselineOutOfLinePrologue` and starts a mark-compact collection, the
58+
conservative root scan walks the half-built baseline `InternalFrame`, and
59+
`ClearStaleLeftTrimmedPointerVisitor` dereferences a junk slot —
60+
`EXC_BAD_ACCESS / KERN_INVALID_ADDRESS at 0xe`, every time, on every machine in
61+
the upstream thread and in this repo's own crash report
62+
(`~/Library/Logs/DiagnosticReports/node-2026-08-20-155211.ips`, pid 77470 —
63+
the exact pid jest reported that afternoon). The trigger population is "lots of
64+
JS compiled through the baseline tier under `vm`" — i.e. jest's module
65+
sandboxing — which is why test runs hit it and the dev server doesn't seem to.
66+
67+
Timing-dependent, so anything that shifts GC pressure shifts the odds: a warm
68+
SWC transform cache makes runs lighter and faster, which is the entire truth
69+
behind "it always works the second time."
70+
71+
## Why it survived — a two-year misattribution
72+
73+
This repo *had* a real native-binding segfault in June 2026: real-SQLCipher
74+
suites running under jsdom, fixed then with `@jest-environment node` docblocks
75+
plus `workerIdleMemoryLimit`. Every SIGSEGV since inherited that diagnosis —
76+
"the known native-SQLCipher worker-teardown flake, run it again." The crash
77+
report that broke the story open showed the dying worker had **no**
78+
`better_sqlite3.node` loaded at all: whatever was crashing, it wasn't the
79+
binding. From there the faulting stack matched nodejs/node#62393
80+
frame-for-frame, down to the `0xe`.
81+
82+
Two lessons worth the ink: a *fixed* bug's ghost can misfile every later bug
83+
that shares its symptom, and macOS keeps the receipts — one `.ips` file in
84+
`DiagnosticReports` ended a months-long guessing game in about a minute.
85+
86+
## How to verify
87+
88+
- Workers carry the flag: while a suite runs,
89+
`ps -axo command | grep -c 'no-sparkplug.*processChild'` counts the fleet.
90+
- Soak: repeated full `npm run test:unit` runs (alternating `npx jest
91+
--clearCache` to recreate the heavier cold-cache condition) with
92+
`~/Library/Logs/DiagnosticReports/node-*.ips` watched for new arrivals.
93+
Upstream's matrices say the same thing at larger n: 0 crashes in 40, 6, and
94+
4 flagged runs against 1-in-5-ish unflagged, at no measurable wall-time cost.
95+
- Retiring it someday: the guard should outlive the upstream bug, not the
96+
repo. When nodejs/node#62393 is actually fixed in the supported Node lines
97+
(verify against the issue, not the changelog), delete `armSparkplugGuard()`
98+
and return the scripts to plain `jest`.

‎jest.config.ts‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -27,18 +27,20 @@ const config: Config = {
2727
coverageProvider: 'v8',
2828
testEnvironment: 'jsdom',
2929
// Recycle a worker once it crosses this resident-memory threshold. Over the
30-
// full ~350-file suite a worker keeps the same process alive across dozens of
31-
// test files; without recycling, memory accumulates and GC timing grows
32-
// aggressive, which is one of the conditions that makes the native SQLCipher
33-
// binding (loaded by the real-binding DB suites) flaky-segfault under parallel
34-
// load. The DB suites also opt into the `node` environment via a per-file
30+
// full suite a worker keeps the same process alive across dozens of test
31+
// files; without recycling, memory accumulates and GC grows aggressive. (The
32+
// real-binding DB suites also opt into the `node` environment via a per-file
3533
// `@jest-environment node` docblock so their native Buffers never cross a
36-
// jsdom realm boundary — the other half of that fix.
34+
// jsdom realm boundary.) Note the long-standing intermittent worker SIGSEGV
35+
// was ultimately neither of those things — it is a V8 Sparkplug GC bug,
36+
// nodejs/node#62393, suppressed by the --no-sparkplug guard in
37+
// jest.global-setup.js and the npm test scripts.
3738
workerIdleMemoryLimit: '512MB',
38-
// Runs once before the whole suite: rebuilds the real SQLCipher binding if it
39-
// was compiled against a different Node ABI than the one running, so the
40-
// real-binding DB suites self-heal after a Node upgrade instead of failing
41-
// with NODE_MODULE_VERSION. Cheap no-op when the ABI already matches.
39+
// Runs once before the whole suite: arms the V8 Sparkplug segfault guard
40+
// (nodejs/node#62393 — see the note in jest.global-setup.js) and rebuilds the
41+
// real SQLCipher binding if it was compiled against a different Node ABI than
42+
// the one running, so the real-binding DB suites self-heal after a Node
43+
// upgrade instead of failing with NODE_MODULE_VERSION.
4244
globalSetup: '<rootDir>/jest.global-setup.js',
4345
// Add more setup options before each test is run
4446
setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],

‎jest.global-setup.js‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
'use strict';
22

3-
// Jest globalSetup — heal a stale native SQLCipher binding BEFORE any suite runs.
3+
// Jest globalSetup — two jobs, both of which must happen BEFORE any suite runs:
4+
// arm the V8 Sparkplug segfault guard (below), and heal a stale native
5+
// SQLCipher binding.
46
//
57
// The real-binding suites (db-backup, graph-integrity, memories-commands, the
68
// migration / content-hash / run-sql-handler suites) load the actual
@@ -68,6 +70,36 @@ function healCopy(copy) {
6870
}
6971
}
7072

73+
// --- V8 Sparkplug segfault guard (nodejs/node#62393) ---
74+
//
75+
// V8 13.6 (Node 24) has a GC race: a stack-guard interrupt fired inside
76+
// Sparkplug's Builtins_BaselineOutOfLinePrologue can start a mark-compact
77+
// whose ClearStaleLeftTrimmedPointerVisitor dereferences a junk frame slot —
78+
// SIGSEGV at address 0xe. Under jest that kills a worker mid-run and fails an
79+
// arbitrary innocent suite ("A jest worker process ... was terminated by
80+
// another process: signal=SIGSEGV") roughly 1 run in 5 on this codebase. It is
81+
// NOT the native SQLCipher binding: crash reports show workers dying with no
82+
// better_sqlite3.node loaded at all. Upstream still reproduces it on Node 26,
83+
// so a Node upgrade is not the fix; `--no-sparkplug` is the workaround the
84+
// nodejs/node#62393 thread converged on (0 crashes across every reporter's
85+
// matrix, no measurable wall-time cost).
86+
//
87+
// jest-worker forks each worker with the parent's `process.execArgv`, which is
88+
// a plain mutable array it reads at fork time — and globalSetup runs in the
89+
// main jest process before any worker forks. Appending the flag here therefore
90+
// disables the baseline compiler in every worker no matter how jest was
91+
// launched (`npx jest`, `--watch`, a single-file `-u` run...). The npm test
92+
// scripts additionally start jest itself under `node --no-sparkplug` so the
93+
// main process and in-band runs are covered too; this push is the safety net
94+
// for ad-hoc invocations that skip the scripts. Remove once the supported Node
95+
// lines ship the V8 fix (verify against the issue above first).
96+
function armSparkplugGuard() {
97+
if (!process.execArgv.includes('--no-sparkplug')) {
98+
process.execArgv.push('--no-sparkplug');
99+
}
100+
}
101+
71102
module.exports = async function globalSetup() {
103+
armSparkplugGuard();
72104
for (const copy of COPIES) healCopy(copy);
73105
};

‎jest.integration.config.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ const createJestConfig = nextJest({
1414
const config: Config = {
1515
coverageProvider: 'v8',
1616
testEnvironment: 'node', // Integration tests should use node environment
17+
// Same globalSetup as the unit config: arms the V8 Sparkplug segfault guard
18+
// (nodejs/node#62393 — see the note in jest.global-setup.js) and heals a
19+
// stale native SQLCipher binding before the real-binding suites load it.
20+
globalSetup: '<rootDir>/jest.global-setup.js',
1721
// Add more setup options before each test is run
1822
setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],
1923
moduleNameMapper: {

‎package-lock.json‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)