v1.2.4: catalog WAL grows to metadata_bytes/4 (1 GiB here) before checkpointing; stage collection writes ~4× the WAL it reclaims #214

Summary

We embed tgrep serve --shared (managed-v2, v1.2.4) in the Copilot runtime. On a large Windows repository, one namespace's catalog.sqlite has not been checkpointed in the 80 minutes since the namespace was created. The database file is still 4,096 bytes, and the whole 189 MiB catalog lives in a 427 MB WAL that keeps growing at about 110–125 MB/h.

Outside storage pressure, tgrep checkpoints only in catalog_io::begin(), with a no-wait attempt once the WAL reaches metadata_bytes / 4. This catalog needs a large metadata_bytes (ours is 4 GiB), so absent storage pressure the WAL will grow to about 1 GiB before its first checkpoint.

Most of the steady-state WAL appears to come from collecting a 1.83 GB build-stage object. Collection truncates members in 64 KiB steps, and each step costs two FULL-sync catalog commits plus a file sync_all. The WAL grows roughly 4× faster than collection reclaims, and collection reclaims about 31 MB/h.

Nothing has failed, and admission has plenty of headroom. The concern is steady-state disk use, read cost, crash-recovery time and reclamation speed.

Environment

• tgrep 1.2.4 (tgrep serve --shared, managed-v2) on Windows 11 Enterprise, NTFS.

• One repository namespace: about 293k files, with a 3.15 GB published generation.

• Our policy differs from the illustrative policy in SHARED_WORKTREE_INDEXES.md. These differences look relevant here:

  Field	Ours	Illustrative
  retention	bounded, 8 GiB target	retain-all
  advancement	adaptive	fixed
  work.workers	1	2
  work.staging_bytes	8 GiB	512 MiB
  work.metadata_bytes	4 GiB	64 MiB
  collection.checkpoint_grace_ms	1 h	5 min
  collection.generation_grace_ms	24 h	1 h
  collection.retry_ms	5000	1000

  Most other work fields also differ; we can share the full policy if that helps. The collection schedule (periodic, 60 s), on_pressure and per-pass bounds match: max_duration_ms 200, max_examined 128, max_removed 16, max_delete_bytes 16 MiB, chunk_bytes 64 KiB and max_pages 16. A 64 MiB metadata_bytes can't hold this catalog, because database page payload is capped at one third of the budget and the database is already 189 MiB.

Observations

File sizes were sampled through shared read handles. Frame counts, commit counts and operation records come from read-only copies of catalog.sqlite and its WAL. Times are local.

• The live database (read from a copy) is 189 MiB, with 1.22 M member_seals rows. The WAL is already about 2.2× the database.
• WAL growth samples ranged from 78 to 198 MB/h. From 17:22:27 to 17:35:33 it averaged about 107 MB/h, so the WAL reaches 1 GiB in roughly 5–6 more hours.
• Between two copies taken 352 s apart, the WAL gained 2,967 frames (12.2 MB) in 525 commits, about 5.7 pages (23 KB) per commit. 39 collection passes were accepted in that window, which works out to about 13.5 commits and 313 KB of WAL per pass. This counts all writers; the catalog's retained operations at the time were 62 collection passes and 2 reconcile passes.

Collection, over the last 62 completed passes (540 s), from the copied operations table:

• Every pass resumed the same build-stage object in pending-deletion: 1,828,781,245 bytes in 358 files. Totals were 62 examined, 0 removed and 0 retired.
• Together they reclaimed 4,605,029 logical bytes (741,376 allocated), about 30.7 MB/h. Extrapolated, the stage takes about 2.5 days to delete, and until then it keeps about 1.8 GB on disk alongside the 3.15 GB generation.
• The WAL grows roughly 3.5–4× faster than collection reclaims (107–125 MB/h against 30.7 MB/h).
• Every pass used its whole 200 ms max_duration_ms budget (elapsed 200–278 ms, average 220 ms). No pass reclaimed more than 196,608 bytes (3 × 64 KiB).
• 33 of the 62 passes reclaimed nothing. Those verified more (on average 7.2 pages and 261 KiB) than the 29 passes that reclaimed bytes (3.1 pages and 155 KiB). In total, verification covered 3,616 proof rows and 9.1 MB to reclaim 4.6 MB, with 1 verification restart, 0 skips and 0 errors.
• Passes started on average every 8.85 s.

Code pointers (v1.2.4)

• configure() sets synchronous=FULL, wal_autocheckpoint=0, mmap_size=0 and a 256 KiB page cache: catalog_io.rs#L50-L64.
• Every catalog mutation goes through catalog_io::begin() (catalog.rs#L593-L612). It attempts a no-wait wal_checkpoint(TRUNCATE) only when the WAL is at least metadata_bytes / 4: catalog_io.rs#L279-L292 and catalog_io.rs#L109-L134.
• Apart from SQLite's own checkpoint when the last connection closes cleanly, the only other production checkpoint is truncate_catalog_wal() (catalog.rs#L652), called after a namespace-storage-allocation refusal: work.rs#L457-L467.
• Collection saves its progress for each examined object, even when it resumes the same object (collection.rs#L2460). Every collection_transaction also rewrites the operation's progress record: collection.rs#L2761-L2801.
• delete_member truncates in chunk_bytes steps (collection.rs#L2894). Each step commits an intent (L2914-L2921), calls set_len and sync_all (L2939-L2940), then commits the credit (L3036-L3037).

Impact

• Disk: each namespace can hold up to metadata_bytes / 4 of WAL before its first checkpoint. Here that is 1 GiB, about 5× the database, and it multiplies across repositories.
• Reads: SQLite notes that read performance "still falls off with increasing WAL file size" (wal.html §2.3). With mmap_size=0 and a 256 KiB cache, more catalog reads go through the wal-index and file reads. We haven't measured this.
• Crash recovery: after an unclean exit, the first opener runs WAL recovery under an exclusive lock (wal.html §9), which rebuilds the wal-index from the WAL. That cost grows with WAL size and sits on the readiness path (see Managed shared mode on a large Windows repo: 25–32 min cold readiness, watcher input churn, scheduler stall, query cliffs (v1.2.3) #210). We haven't measured this.
• First checkpoint: the whole catalog currently exists only in the WAL, so the first checkpoint will copy all of it inside one mutation's begin(). TRUNCATE checkpoints block new writers while they run, and the other writers' busy timeout is WRITER_WAIT (250 ms). We haven't measured this stall.
• Reclamation: a 1.8 GB stage takes about 2.5 days to delete at the observed rate.

Suggestions and questions

These keep the documented rule that "WAL truncation is attempted without waiting before catalog mutations, never after a successful commit" (SHARED_WORKTREE_INDEXES.md#L1793-L1794).

1. Decouple the checkpoint threshold from metadata_bytes. metadata_bytes has to be sized for the largest catalog, so metadata_bytes / 4 grows with it. Options include a cap such as min(metadata_bytes / 4, 64 MiB), a threshold relative to the live database size, or a separate policy field. Checkpoints would still be attempted at the same point, before a mutation and without waiting, just with a smaller WAL.
2. Fewer commits per reclaimed chunk. Could the per-object progress save share the first intent's transaction? Could chunk N's credit share a transaction with chunk N+1's intent?
3. Chunk size and cadence. On NTFS, the cost of a truncation doesn't depend on its length. With 64 KiB steps, each pass appears to be bound by flush latency; we infer this from the cap of 3 steps per 200 ms pass, but haven't profiled it. Is there a constraint that keeps chunk_bytes small, such as the crash window, authentication-block alignment or verification granularity? What chunk_bytes, max_duration_ms, retry_ms and metadata_bytes would you recommend for repositories this size?
4. Verification-only passes. Is it expected that about half the passes verify without reclaiming anything? Could that verification carry over, so the next pass truncates immediately?

You know the invariants much better than we do. We're happy to test a build or collect more data. We can also tune our own retry_ms, chunk_bytes and metadata_bytes, and would welcome guidance on safe values.

Related

• Managed shared mode on a large Windows repo: 25–32 min cold readiness, watcher input churn, scheduler stall, query cliffs (v1.2.3) #210: cold readiness on v1.2.3.
• v1.2.4: ready view starves reconcile/periodic verification while the only work slot is reserved under storage pressure #212: reconcile starvation under storage pressure on v1.2.4.