Skip to content

multiprocess_wal: empty WAL + leftover -tshm fails open with short read on WAL frame #8536

Open

Description

@berenddeboer

Summary

Opening a database with experimental: ["multiprocess_wal"] fails when the -wal file is empty (0 bytes) but a leftover -tshm file still claims WAL frames:

failed to open database …: I/O error: short read on WAL frame at offset 3773952: expected 4096 bytes, got 0

The main SQLite file is intact (PRAGMA integrity_check = ok). Deleting -wal and -tshm lets Turso open it again. Stock sqlite3 also opens the main file because it ignores -tshm.

This is the same error class as #6454, but it is native @tursodatabase/database + multiprocess_wal, not wasm open/close.

Environment

  • @tursodatabase/database 0.7.2
  • Bun 1.4.0 (also observed from our Effect layer using the same native package)
  • Linux x86_64

Minimal reproduction

import { connect } from "@tursodatabase/database";
import { existsSync, statSync, truncateSync } from "node:fs";

const path = "./repro.db";
const wal = path + "-wal";
const tshm = path + "-tshm";

const db = await connect(path, { experimental: ["multiprocess_wal"] });
await db.exec("CREATE TABLE t (id INTEGER PRIMARY KEY, blob BLOB)");
const stmt = db.prepare("INSERT INTO t (blob) VALUES (?)");
const blob = Buffer.alloc(4096, 7);
for (let i = 0; i < 900; i++) {
  await stmt.run(blob);
}
await db.close();

console.log({
  db: statSync(path).size,
  wal: existsSync(wal) ? statSync(wal).size : -1,
  tshm: existsSync(tshm) ? statSync(tshm).size : -1,
});
// close() does not checkpoint: WAL stays large, -tshm magic is "TSHMWAL"

truncateSync(wal, 0);

await connect(path, { experimental: ["multiprocess_wal"] });

Observed on this run:

  • after close: db 2854912, wal 3786312, tshm 86016 (TSHMWAL\0)
  • after truncateSync(wal, 0): reopen fails with short read on WAL frame at offset 3773952: expected 4096 bytes, got 0

Offset 3773952 = 921 * 4096.

What we hit in the wild

A local multi-process runtime (auth / graphql / job worker sharing one file via multiprocess_wal) refused to open:

  • pf.db 2281472 bytes (557 pages × 4096), SQLite 3.x, WAL journal mode, integrity ok
  • pf.db-wal 0 bytes
  • pf.db-tshm 86016 bytes, header magic TSHMWAL
  • error: short read on WAL frame at offset 3345472: expected 4096 bytes, got 0 (3345472 = 817 * 4096)

We do not know what truncated that WAL (process kill, checkpoint, or another opener creating an empty -wal over a leftover -tshm). The synthetic truncate above matches the failure exactly.

Expected

If -wal is empty or shorter than the -tshm frame index, treat that as “no WAL frames” (or discard stale -tshm) and open the checkpointed main file. Do not fail connect().

SQLite recovers from a missing/empty WAL; Turso’s multiprocess WAL index currently bricks open instead.

Workaround

Delete -wal and -tshm, then reopen. Uncheckpointed WAL data is already gone when -wal is 0 bytes; the main file is usable.

Activity

  1. added theissue type on Aug 28, 2026
  2. penberg commented on Sep 22, 2026

    @penberg
    Collaborator

    @roboturso triage

  3. roboturso commented on Sep 22, 2026

    @roboturso
    Contributor

    @roboturso reproduced this issue at commit ca8933a2e32b74edf6e24650c4f6ace9f0ad6f3c.

    Reproduce:

    cargo test -p turso_core --lib multiprocess_empty_wal_stale_tshm_tests::multiprocess_open_recovers_when_wal_is_empty_but_tshm_claims_frames -- --exact

    Reproducer: https://github.com/roboturso/turso/blob/issue-8536/REPRODUCER-8536.md

  4. v5vsvkmwpv-debug commented on Oct 25, 2026

    @v5vsvkmwpv-debug

    We hit this exact failure in production and can report what triggers it — the issue says the cause of the truncation was unknown.

    Trigger: OS crash / power-off. In our case a single power-off on 2026-10-18 20:25 left 9 databases in the store simultaneously in this state (-wal 0 bytes, leftover -tshm 86016 bytes, magic TSHMWAL). All 9 failed connect() with this error class; the offset varied per DB (e.g. 366712, 3773952).

    Still present in 0.8.3 — upgrading does not fix it.

    Environment

    • @tursodatabase/database 0.8.3 (latest)
    • macOS 14 arm64, Node 26.7.0
    • native binding, experimental: ["encryption", "multiprocess_wal"]

    Two additions to the workaround

    1. The -shm sidecar may need to go too. Your workaround says delete -wal and -tshm. In our case a stale stock-sqlite3 -shm was also present and had to be moved as well — all three sidecars were quarantined together. Leaving -shm behind left the DB still failing.

    2. Do not use the stock sqlite3 CLI to inspect a live Turso DB. Opening one with sqlite3 recreates -shm, which breaks Turso's -tshm correspondence and turns a previously healthy DB into one that fails — with this error, or with Corrupt database: Invalid page type: 0 depending on what the stale index ends up pointing at. This bit us during diagnosis: it made the blast radius look bigger than it was. Raw file inspection (stat/od) is safe; opening the DB is not.

    The file-level signature is not authoritative

    We found DBs with the same "stale signature" (-tshm present, -wal 0 bytes) that Turso opened fine. The authoritative check is a driver connect() attempt, not the file state.

    Scale matters when reproducing

    A small WAL truncated to 0 with a leftover -tshm may open successfully (SQLite's checksum recovery silently drops the tail). We needed ~900 rows × 4 KiB (≈3.7 MiB WAL) for a deterministic failure — matching your offset 3773952 = 921 × 4096 observation.

    Self-healing tool

    We built one and have been running it on a schedule since: it classifies every DB in the store via the real driver, quarantines only DBs that are provably stale and have an empty -wal, re-verifies, and rolls the sidecars back if the re-open still fails. No data loss (row counts identical before/after).

    Source: https://github.com/v5vsvkmwpv-debug/opencode-config/blob/main/bin/turso-wal-check.sh

    Happy to provide the repro script or more data if useful. Related: #6454, #8593.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions