Add snapshot SQLite stress harness#94967
Conversation
|
Codex review: found issues before merge. Reviewed July 1, 2026, 3:37 PM ET / 19:37 UTC. Summary PR surface: Tests +76, Config +3, Other +712. Total +791 across 4 files. Reproducibility: yes. for the PR finding: source inspection gives a high-confidence path, run smoke against a reused state dir and then default or large against the same dir. The second run restores older batches with a different row count and validates them against only the current profile's Review metrics: 2 noteworthy metrics.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Scope restored invariant checks to the current run or persist per-batch expected row counts, add a focused cross-profile reused-state-dir regression, and merge only after the upstream snapshot command contract is accepted or adopted. Do we have a high-confidence way to reproduce the issue? Yes for the PR finding: source inspection gives a high-confidence path, run smoke against a reused state dir and then default or large against the same dir. The second run restores older batches with a different row count and validates them against only the current profile's Is this the best way to solve the issue? No: the harness is in the right scripts/package-script boundary, but the reusable-state-dir invariant should be scoped to current-run batches or explicit per-batch metadata. Keeping the current global batch-count check would make a supported reuse path report false partial transactions. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 24d94a54a516. Label changesLabel justifications:
Evidence reviewedPR surface: Tests +76, Config +3, Other +712. Total +791 across 4 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (1 earlier review cycle)
|
ce28ab4 to
ff7dd74
Compare
ff7dd74 to
764107b
Compare
764107b to
14eacb6
Compare
14eacb6 to
adac64c
Compare
adac64c to
bf54af8
Compare
|
@clawsweeper re-review Addressed the worker lifecycle finding at What changed:
Validation passed:
|
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Updated PR #94967 again at Additional fixes:
Validation passed:
|
|
🦞🧹 I asked ClawSweeper to review this item again. |
868ee86 to
8701065
Compare
8701065 to
50bc0b1
Compare
|
@clawsweeper re-review Updated PR #94967 to current head Post-rebase validation passed:
|
|
🦞🧹 I asked ClawSweeper to review this item again. |
af2abc2 to
409d541
Compare
|
Thanks, Gio. The useful cross-process SQLite stress-harness direction from this PR has now been adapted to the current snapshot/state APIs and landed in #105740: We could not land this branch in place because Your contribution is preserved in the landed commit with:
Closing this PR as superseded by the landed canonical implementation. |
Depends on #94805.RFC: openclaw/rfcs#20Related: #94675, #94646## Summary- Adds an opt-in SQLite snapshot stress harness for the core
snapshotwork.- Exercises snapshot create/copy/restore while a concurrent writer commits transaction batches into the same source SQLite database.- Covers both Phase 1 DB units:--target globaland--agent <id>.- Keeps writer lifecycle monitoring attached for the full snapshot loop so worker result/error/exit states fail deterministically instead of being missed.- Supports repeated runs against the same--state-dirby continuing batch IDs and using run-scoped sync/restore scratch paths.- Adds package scripts for smoke/default/large profiles.## ScopeThis is measurement and validation plumbing only. It does not add WAL bundles, scheduling, object storage, failover orchestration, or CI-gated long stress runs.## Real behavior proofBehavior or issue addressed:Core snapshot artifacts need a repeatable stress harness that proves restored safe-sync files remain SQLite-valid and do not expose partial writer transactions while the source DB is being mutated. This update also addresses review findings by keeping a single writer monitor attached from start through stop and making documented--state-dirreuse repeatable.Real environment tested:Windows checkout atgiodl/snapshot-stress-pr5(50bc0b102e), stacked on PR #94805 head86f92a1e70, Node v22.22.3, source harness usingnode:sqlite, OpenClaw local SQLite state databases, and the stacked core snapshot provider from this PR stack. Node prints the expected experimental warning fornode:sqlite.Exact steps or command run after this patch:1.node scripts/run-vitest.mjs run test/scripts/bench-snapshot-sqlite.test.ts2.pnpm exec oxlint scripts/bench-snapshot-sqlite.ts test/scripts/bench-snapshot-sqlite.test.ts3.pnpm exec oxfmt --check scripts/bench-snapshot-sqlite.ts scripts/bench-snapshot-sqlite.mjs test/scripts/bench-snapshot-sqlite.test.ts package.json4.git diff --check5. Repeat-run proof: runnode --import tsx scripts/bench-snapshot-sqlite.mjs --profile smoke --state-dir <same-dir> --output <one.json>, then run the same command again with<two.json>.6.pnpm test:snapshot:stress7.pnpm test:snapshot:stress:large8.node --import tsx scripts/bench-snapshot-sqlite.mjs --profile default --agent main --output .artifacts/snapshot-stress/agent-default.json9.node --import tsx scripts/bench-snapshot-sqlite.mjs --profile large --agent main --output .artifacts/snapshot-stress/agent-large.jsonEvidence after fix:Focused validation passed:textTest Files 1 passed (1)Tests 5 passed (5)Duration 900msRepeated--state-dirsmoke proof passed twice in the same directory:textSNAPSHOT_STRESS_PROFILE=smokeSNAPSHOT_STRESS_TARGET=globalSNAPSHOT_STRESS_ITERATIONS=3SNAPSHOT_STRESS_RESTORES_VERIFIED=3SNAPSHOT_STRESS_WRITER_ROWS=32SNAPSHOT_STRESS_SNAPSHOT_P95_MS=25.832SNAPSHOT_STRESS_RESTORE_P95_MS=9.431SNAPSHOT_STRESS_SNAPSHOT_BYTES_MAX=962560SNAPSHOT_STRESS_WAL_BYTES_AFTER=0--- second ---SNAPSHOT_STRESS_PROFILE=smokeSNAPSHOT_STRESS_TARGET=globalSNAPSHOT_STRESS_ITERATIONS=3SNAPSHOT_STRESS_RESTORES_VERIFIED=3SNAPSHOT_STRESS_WRITER_ROWS=32SNAPSHOT_STRESS_SNAPSHOT_P95_MS=25.595SNAPSHOT_STRESS_RESTORE_P95_MS=9.664SNAPSHOT_STRESS_SNAPSHOT_BYTES_MAX=962560SNAPSHOT_STRESS_WAL_BYTES_AFTER=0Global default proof:textSNAPSHOT_STRESS_PROFILE=defaultSNAPSHOT_STRESS_TARGET=globalSNAPSHOT_STRESS_ITERATIONS=20SNAPSHOT_STRESS_RESTORES_VERIFIED=20SNAPSHOT_STRESS_WRITER_ROWS=736SNAPSHOT_STRESS_SNAPSHOT_P95_MS=22.898SNAPSHOT_STRESS_RESTORE_P95_MS=10.462SNAPSHOT_STRESS_SNAPSHOT_BYTES_MAX=995328SNAPSHOT_STRESS_WAL_BYTES_AFTER=0Global large proof:textSNAPSHOT_STRESS_PROFILE=largeSNAPSHOT_STRESS_TARGET=globalSNAPSHOT_STRESS_ITERATIONS=100SNAPSHOT_STRESS_RESTORES_VERIFIED=100SNAPSHOT_STRESS_WRITER_ROWS=7648SNAPSHOT_STRESS_SNAPSHOT_P95_MS=22.478SNAPSHOT_STRESS_RESTORE_P95_MS=14.069SNAPSHOT_STRESS_SNAPSHOT_BYTES_MAX=1290240SNAPSHOT_STRESS_WAL_BYTES_AFTER=0Agent default proof:textSNAPSHOT_STRESS_PROFILE=defaultSNAPSHOT_STRESS_TARGET=agent:mainSNAPSHOT_STRESS_ITERATIONS=20SNAPSHOT_STRESS_RESTORES_VERIFIED=20SNAPSHOT_STRESS_WRITER_ROWS=528SNAPSHOT_STRESS_SNAPSHOT_P95_MS=16.522SNAPSHOT_STRESS_RESTORE_P95_MS=6.868SNAPSHOT_STRESS_SNAPSHOT_BYTES_MAX=147456SNAPSHOT_STRESS_WAL_BYTES_AFTER=0Agent large proof:textSNAPSHOT_STRESS_PROFILE=largeSNAPSHOT_STRESS_TARGET=agent:mainSNAPSHOT_STRESS_ITERATIONS=100SNAPSHOT_STRESS_RESTORES_VERIFIED=100SNAPSHOT_STRESS_WRITER_ROWS=5248SNAPSHOT_STRESS_SNAPSHOT_P95_MS=15.794SNAPSHOT_STRESS_RESTORE_P95_MS=8.328SNAPSHOT_STRESS_SNAPSHOT_BYTES_MAX=348160SNAPSHOT_STRESS_WAL_BYTES_AFTER=0Observed result after fix:The harness created snapshots while a writer committed transaction batches, copied each snapshot to a simulated synced location, restored every copied artifact to a separate SQLite file, verifiedPRAGMA integrity_check, and rejected partial batch visibility. The current Windows default/large runs verified 240 restores across global and per-agent targets with 13,160 committed writer rows total. The writer lifecycle monitor now stays attached throughout the snapshot loop, and repeated--state-dirruns no longer collide on stress table batch IDs or sync/restore output paths.Supporting validation passed:-node scripts/run-vitest.mjs run test/scripts/bench-snapshot-sqlite.test.ts-pnpm exec oxlint scripts/bench-snapshot-sqlite.ts test/scripts/bench-snapshot-sqlite.test.ts-pnpm exec oxfmt --check scripts/bench-snapshot-sqlite.ts scripts/bench-snapshot-sqlite.mjs test/scripts/bench-snapshot-sqlite.test.ts package.json-git diff --checkWhat was not tested:Cross-platform stress, object-storage sync, WAL bundle/delta behavior, and production failover orchestration are intentionally left for later validation. Local Crabbox/Testbox proof was not run because this Windows/WSL machine does not currently have the Crabbox wrapper binary available in the expected WSL locations.Post-rebase validation on 50bc0b1:- ode scripts/run-vitest.mjs run test/scripts/bench-snapshot-sqlite.test.ts passed 1 file / 5 tests.- pnpm exec oxlint scripts/bench-snapshot-sqlite.ts test/scripts/bench-snapshot-sqlite.test.ts passed.- pnpm exec oxfmt --check scripts/bench-snapshot-sqlite.ts scripts/bench-snapshot-sqlite.mjs test/scripts/bench-snapshot-sqlite.test.ts package.json passed.- git diff --check giodl/snapshot-restore-proof-pr4...HEAD passed.- pnpm test:snapshot:stress:smoke passed on the rebased stack.Post-rebase refresh on
50bc0b102eover #94805 head86f92a1e70:node scripts/run-vitest.mjs run test/scripts/bench-snapshot-sqlite.test.tspassed 1 file / 5 testspnpm exec oxlint scripts/bench-snapshot-sqlite.ts test/scripts/bench-snapshot-sqlite.test.tspassedpnpm exec oxfmt --check scripts/bench-snapshot-sqlite.ts scripts/bench-snapshot-sqlite.mjs test/scripts/bench-snapshot-sqlite.test.ts package.jsonpassedgit diff --check giodl/snapshot-restore-proof-pr4...HEADpassedpnpm test:snapshot:stress:smokepassed with 3 snapshots/restores verified, 32 writer rows, snapshot p95 25.814 ms, restore p95 9.590 ms, max artifact bytes 962560, WAL bytes after 0codex review --base origin/giodl/snapshot-restore-proof-pr4returned no discrete correctness issue