Base: pingdotgg/t3code 66ee628e0875b2b735b199f42e6151e347833637.
Reviewed head: 9201840474a9621e785b268c8ae2d779c2fdee5a (PR #24145).
Environment: macOS arm64, APFS SSD, Node 24.17.0, SQLite 3.53.0, 4 KiB pages, Codex CLI 0.164.0.
All databases and provider workspaces are disposable. No live T3 database was opened read-write or copied.
EventSink.write already commits arrays atomically, but provider ingestion called it separately for each delivered event. Stream.mapEffect in the session subscription also flattened queued bursts into singletons. Stable projection snapshots assigned indexed columns unnecessarily, causing SQLite to rewrite index entries. Stale/equal thread.visit commands emitted another full snapshot.
The patch preserves queued chunks and commits already-available root-run message/item snapshots in order, with at most 32 normalized events and 1 MiB normalized JSON per transaction. A larger individual event commits alone. There is no timer, waiting window, or event coalescing. Chunk boundaries, controls, changed nodes, and held-item position reservations flush pending snapshots; failure/cancellation drains the accepted prefix. Root snapshot groups hold the existing thread command lock so Stop cannot settle their work before the drain. Control events and child/post-terminal work retain their existing ingestion paths. No schema, wire protocol, durability pragma, checkpoint policy, or command acknowledgment changes.
Unchanged node repeats do not split a batch. Changed nodes commit immediately because their next deduplication reads projections. Projection updates avoid assigning stable indexed keys and retain the existing upsert for inserts/key changes. No-op visits retain command receipts and emit no domain event.
Forward visits still have a full shell payload on this base. Open maintainer PR #21778 owns the compact visit/PR-refresh payload and compatible client transport work; this patch does not duplicate it. Open #23211 owns removal of the unused correlation index.
Setup and commands: LINUX-REPRO.md.
The Linux extension uses Node 26.12.1 / SQLite 3.54.0, Linux aarch64 kernel 7.2.12-orbstack-00394-gbecdd6130068, and a dedicated 64 GiB btrfs loopback filesystem with COW enabled. Data is single-copy; metadata uses the default DUP profile; compression is off. Docker's native btrfs mount has nodatacow, so it was not used as the database filesystem for these comparisons. The container has an 8 GiB memory limit on a ~12 GiB VM. This is a virtualized storage stack on an APFS/SSD host, not a direct NVMe measurement.
linux-seed.mjs creates a 22,010,114,048-byte database with 1,288,000 synthetic historical event rows and the real schema/indexes. Each measured variant starts from a full, non-reflink copy of the same seed. The seed exercises a large event table and indexes; it does not reconstruct the reporter's complete projection/history shape. Four concurrent root runs then persist 2,400 command snapshots plus 16 final lifecycle events. Logical UTF-8 payload+metadata is 2,339,400 bytes in every preserved-event comparison.
The paced source delivers two snapshots per second per run for five minutes. The burst source delivers the same snapshots in available groups of eight at accelerated throughput. Persistence.layerFromPath provides the real reader and SQLite checkpoint workers, NORMAL synchronous mode, 1,000-frame checkpoint threshold, 32 MiB journal-size limit, and existing admission/backpressure behavior. The external provider and Git checkpoint operation are stubbed; SQLite checkpointing is not. No SQLite durability setting was relaxed for measurement.
| Workload | Node process write_bytes before → after | Write/logical ratio before → after | Reduction |
|---|---|---|---|
| Five minutes, singleton delivery | 269,357,056 → 183,468,032 | 115.1× → 78.4× | 31.9% |
| Same data, available groups of eight | 282,013,696 → 45,584,384 | 120.5× → 19.5× | 83.8% |
All four variants have identical per-thread event-row hashes and final-projection hashes. Global scheduling between independent threads and page placement may differ. The singleton CPU optimization from the CodeRabbit review was separately replayed against the same large seed: the 256-snapshot event/projection hashes were unchanged and temporary-file writes stayed at zero. Storage measurements include the statement-journal fix; the paced run predates only that later redundant-encode removal.
strace -ff records successful write/pwrite and sync calls from the Node process and all worker threads. For the paced case, WAL pwrite bytes fall from 223,169,704 to 157,817,816; database checkpoint pwrite bytes fall from 22,982,656 to 16,531,456. Other traced writes fall from ~13.05 MB to ~14 KB after removing statement-journal spills. These syscall totals differ from Linux write_bytes, which accounts for storage I/O rather than simply summing every write request (kernel accounting reference).
A final explicit checkpoint and btrfs filesystem sync drain the tail after each timed workload; natural-window and drained counters are both retained. The paced database-filesystem loop-device counter is 278,724,608 → 204,972,032 bytes; it includes that filesystem's metadata/writeback but excludes SQLite's temporary directory on the native VM filesystem. It is neither a whole-machine device counter nor physical SSD/NVMe writes. Setup/copy/startup I/O is recorded separately and excluded from the steady-state delta. Tracer output is written outside the measured database filesystem. No global cache drop or shared VM shutdown was performed. Runtime durations are single-run observations, not latency claims.
fullserver-pairs.mjs starts the actual bin.ts server with fresh isolated state, authenticates a Node WebSocket client, creates four threads, and sends 2,000 visit commands. It verifies each thread's watermark through RPC and reads the HTTP environment descriptor. A private measurement-only Unix socket performs checkpoints inside the server process before/after the window so its final checkpoint writes are included. Production background checkpointing remains active. This is a separate small-database server test, not a live-provider run against the 22 GB seed.
Final paired run: 294,830,080 → 93,753,344 process write bytes (68.2% less). Both retain all 2,000 command receipts and the same four visited watermarks. Stored thread.visited events are 2,000 → 4, intentionally omitting unchanged visits. Both databases pass integrity_check. Write counters vary with checkpoint timing; earlier passes also showed substantial savings, rather than identical byte totals.
SQL/syscall correlation locates the unnamed temporary writes inside the single-event INSERT ... RETURNING sequence. A 256-snapshot diagnostic on the large seed writes 1,193,100 temporary bytes with the original insert. Both temp_store=MEMORY and single-row INSERT OR FAIL with the original temp_store setting reduce those writes to zero, with identical events/projections. The latter is the implemented fix. It avoids changing memory behavior for unrelated sorts/temporary tables. SQLite documents alternative conflict algorithms as avoiding statement journals (statement-journal reference).
Every agent-event INSERT writes one row. The surrounding EventSink transaction still rolls back the whole batch on failure; constraints still raise an error. Focused tests verify that a duplicate insert preserves the existing event and next sequence, and that an event-insert failure rolls back events, projections, command receipts, and outbox effects. Global temp_store and durability pragmas remain unchanged.
crash-recovery.mjs uses real EventSink transactions and a small disposable COW-btrfs database. A small page cache forces uncommitted WAL writes, then the parent kills only the child PID it spawned. Results:
- Kill inside a transaction: eight prior committed events survive; none of the eight pending events or projections survive.
- Kill after commit: all sixteen events and projections survive.
- Truncate the final WAL frame by one byte after termination: the prior eight-event transaction survives and the complete last transaction is discarded.
All three databases pass integrity_check. These are process-death and torn-WAL fault tests, not an actual power-cut test. NORMAL mode's existing OS/power-failure semantics still apply (SQLite WAL durability reference). The shared host/VM was not powered off.
Full counters, hashes, failure milestones and source revisions are in deep-validation-results.json. The final 22,013,931,520-byte database, containing 1,290,436 event rows, also passed full integrity_check. No production database, provider credentials, private server logs or raw conversation captures are published.
controller.mjs runs the real RunExecutionService, normalizer, event sink, projection store and migrated schema. Four concurrent runs produce 300 command snapshots each, reusing 16 items per run, against 40,000 seeded historical events. Only the provider runtime and Git checkpoint capture are stubbed. Available source chunks contain 1, 2, 8 or 32 snapshots. This is an accelerated throughput workload, not a claim about ordinary live queue sizes.
| Available snapshots | Commits before → after | WAL frames before → after | WAL reduction | WAL/logical bytes before → after |
|---|---|---|---|---|
| 1 | 1,204 → 1,204 | 27,259 → 19,693 | 27.8% | 95.7 → 69.1 |
| 2 | 1,204 → 604 | 27,112 → 11,351 | 58.1% | 95.2 → 39.8 |
| 8 | 1,204 → 156 | 27,065 → 4,399 | 83.7% | 95.0 → 15.4 |
| 32 | 1,204 → 44 | 27,079 → 2,051 | 92.4% | 95.1 → 7.2 |
Every variant stores the same 1,200 snapshots plus the same 16 final lifecycle events. Logical payload+metadata is 1,173,752 bytes. Per-thread stored event rows and measured final projection tables hash identically before/after. Global inter-thread ordering and page allocation may vary with scheduling. Runtime durations are single-run observations on a shared machine, not latency claims.
live-controller.mjs ran the real Codex app-server adapter and RunExecutionService with an isolated CODEX_HOME/workspace. It requested 40 lines of original SQLite prose without tools/files. The run delivered 537 provider events and stored 49 events over 55.958 seconds (~0.88 persisted events/s), ending in waiting. A queue between adapter and controller models session delivery; actual session chunk/failure behavior is tested separately. The CLI auth copy was removed in finally.
controller-trace.mjs replays those exact raw source chunks and source-pull timestamps against 40,000 historical event rows. Original main: 46 commits, 1,145 frames / 4,717,432 WAL bytes. Patched: 45 commits, 941 frames / 3,876,952 bytes, 17.8% less WAL. Logical payload+metadata is 187,855 bytes, so WAL/logical is 25.1 → 20.6. Stored events and final projections hash identically. Most delivered live text was filtered by the existing streaming policy; eligible persisted updates were mostly singletons, so this trace does not claim the synthetic burst gains. The original live database had different seeding/allocation and produced 885 frames; only paired replay counters are compared.
The earlier projection-only study remains in results.json: 1,200 single-event commits, four threads, 4,000 rows in each projection table and 40,000 event rows (~46 MB), with 12–19% less WAL for stable streaming updates and unchanged WAL for insertion/key-move controls. A separate 107.447-second direct-adapter trace produced 527 persisted events (~4.9/s); paired replay fell from 13,707 to 11,112 frames (18.9%), with identical event/projection hashes. That earlier trace bypasses RunExecutionService's filtering and is not the end-to-end batching result.
The earlier macOS WAL-frame experiments measure frames with automatic checkpoints disabled to avoid frame reuse, after seed/schema writes are checkpointed and truncated. Each frame has a 4,096-byte page and 24-byte header; nonzero database-size markers count commits. The earlier projection benchmark also performs an explicit PASSIVE checkpoint and records backfilled frames/distinct destination pages. Production checkpoint scheduling is unchanged.
Logical bytes here mean UTF-8 payload_json plus metadata_json for measured stored events. WAL bytes are not process-tree write_bytes, checkpoint write bytes, filesystem writes or physical device writes. The earlier macOS runs exclude startup maintenance, logs and the full server/checkpoint worker. The Linux extensions below measure process I/O and production SQLite checkpointing separately. The reporter's exact hardware/provider workload and a physical power interruption remain unverified. No UI or remote/tunnel experiment was run.
Other measured options were rejected: a partial command-id index saved another 3–6% of WAL but requires scanning/rebuilding a large event index at startup; removing the generic stream index breaks explicit query hints in current/older binaries. Large linked-PR shells added about one WAL page per update in the isolated experiment, rather than rewriting every unchanged overflow page. The Linux extension did reproduce statement-journal spills. A global temp_store=MEMORY change was rejected in favor of single-row INSERT OR FAIL, which removed those spills while preserving the transaction rollback boundary. These experiments do not warrant a broader storage refactor.
Save the attached scripts under /tmp/t3-event-write-24003 and run from the patched repository root with dependencies installed. The baseline loader substitutes only the two files below from the pinned base. Synthetic replay does not need provider credentials.
mkdir -p /tmp/t3-event-write-24003
git show 66ee628e0875b2b735b199f42e6151e347833637:apps/server/src/orchestration-v2/ProjectionStore.ts > /tmp/t3-event-write-24003/ProjectionStore.before.ts
git show 66ee628e0875b2b735b199f42e6151e347833637:apps/server/src/orchestration-v2/RunExecutionService.ts > /tmp/t3-event-write-24003/RunExecution.before.ts
vp exec node /tmp/t3-event-write-24003/controller.mjs baseline-controller
vp exec node /tmp/t3-event-write-24003/controller.mjs controller-final
# Optional authenticated live capture, then paired raw replay:
vp exec node /tmp/t3-event-write-24003/live-controller.mjs
vp exec node /tmp/t3-event-write-24003/controller-trace.mjs baseline-live-trace
vp exec node /tmp/t3-event-write-24003/controller-trace.mjs final-live-trace
# Earlier projection-only workloads:
vp exec node /tmp/t3-event-write-24003/benchmark.mjs baseline-final
vp exec node /tmp/t3-event-write-24003/benchmark.mjs after-finalEach script prints its disposable directory; remove it after inspecting the data. A fresh provider turn produces different events; compare before/after using the same capture. Captures are not published. batch-results.json contains the controlled counters/hashes; results.json retains the earlier projection-only counters.
Before the projection fix, the three file-backed regressions write 5/4/6 WAL pages for a small stable node/message/item update; after, one leaf page each. Semantic regressions compare updates to fresh inserts across independent indexed-key, NULL, lifecycle/type, failed-error-index and large-ordinal transitions. Removing a representative guard from each path makes those tests fail (mutations.json).
Queued-batch tests cover singleton immediacy, source pauses, count/byte limits, unchanged/changed nodes, control ordering, accepted-prefix draining on normalization failure/cancellation, and Stop ordering. Both Stop tests fail when the thread lock is removed and pass restored. Session tests verify preserved queued chunks and the valid prefix before failure. The stale/equal visit regression fails on main.
The final check:changed run passed all 290 adjacent tests. Codex/Claude/OpenCode/restart recovery replays: 88 passed. vp run check:changed covers formatting, lint, both Knip passes, GraphQL, affected typechecks and adjacent tests. No repository-wide suite was run.
Model/harness: GPT-6.1-Sol through Codex in T3 Code.