Skip to content

refactor: flip sessions and transcripts to sqlite storage#96625

Closed
jalehman wants to merge 292 commits into
openclaw:mainfrom
jalehman:clawdbot-d02.1.9.1.40/path3-sqlite-storage-flip
Closed

refactor: flip sessions and transcripts to sqlite storage#96625
jalehman wants to merge 292 commits into
openclaw:mainfrom
jalehman:clawdbot-d02.1.9.1.40/path3-sqlite-storage-flip

Conversation

@jalehman

@jalehman jalehman commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Completes the session/transcript storage flip tracked by #88838 by making per-agent SQLite the canonical runtime store for session metadata and transcript events. Legacy sessions.json and active transcript JSONL files are now doctor import inputs or archive artifacts, not runtime fallback state.

Why This Change Was Made

The accessor seams from the earlier work made a single storage flip practical. Shipping a mixed file/SQLite runtime would let session metadata and transcript rows drift, so this PR moves import, runtime access, retention, SDK cleanup, and ratchets together behind the permanent accessor boundary.

User Impact

Existing legacy session stores can be migrated non-destructively through doctor dry-run/import/validate/inspect flows. Runtime session creation, resume, transcript append, compaction, and maintenance use SQLite rows; retention still exports archived JSONL artifacts before deleting rows so support and archive workflows keep inspectable artifacts.

Evidence

  • node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.test.src.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/test-src.tsbuildinfo
  • node scripts/check-session-accessor-boundary.mjs
  • node scripts/run-vitest.mjs src/config/sessions/session-accessor.test.ts src/config/sessions/targets.test.ts src/commands/doctor-session-sqlite.test.ts test/scripts/check-session-accessor-boundary.test.ts src/plugin-sdk/session-store-runtime.test.ts src/config/sessions/session-accessor.conformance.test.ts src/config/sessions/store.session-lifecycle-mutation.test.ts src/plugins/contracts/host-hooks.contract.test.ts src/plugins/contracts/session-entry-projection.contract.test.ts src/plugins/contracts/plugin-sdk-runtime-api-guardrails.test.ts
  • node scripts/run-oxlint.mjs src/config/sessions/session-accessor.test.ts src/config/sessions/targets.ts src/config/sessions/targets.test.ts src/commands/doctor-session-sqlite.test.ts
  • .agents/skills/autoreview/scripts/autoreview --mode local clean after two accepted findings were fixed.
  • Copied local OpenClaw state proof under /tmp/openclaw-sqlite-flip-final-20260625T013042Z: doctor dry-run/import/validate/inspect on codex-proof and codex-image-proof reported zero issues; import wrote 2 entries plus 15 and 18 transcript events respectively.
  • Isolated runtime smoke created an agent SQLite DB, created/resumed a session, appended transcript rows, compacted 4 rows to 3, and archive-before-delete cleanup removed 1 lifecycle entry after writing 1 archive artifact.

@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes scripts Repository scripts commands Command implementations labels Jun 25, 2026
@openclaw-barnacle openclaw-barnacle Bot added size: XL maintainer Maintainer-authored PR labels Jun 25, 2026
@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed June 30, 2026, 2:28 PM ET / 18:28 UTC.

Summary
The PR flips active session metadata and transcript runtime storage from legacy JSON/JSONL files to per-agent SQLite, adds doctor/startup migration tooling, updates runtime/plugin callers, and refreshes docs/tests/proof harnesses.

Reproducibility: yes. from source inspection: explicit sessionFile targets remain valid, but the PR run path now invokes a SQLite-only runtime transcript truncation helper that returns without truncating non-SQLite targets; no tests were run because this was a read-only review.

Review metrics: 3 noteworthy metrics.

  • Persistent schema tables: 7 added. The PR changes the per-agent stored data model, so maintainers need upgrade and migration confidence beyond green unit tests.
  • Plugin runtime session helpers: 4 removed. Removing deprecated runtime helper methods changes a plugin-facing contract and needs explicit compatibility acceptance.
  • Doctor migration mode: 1 added. A new operator-facing SQLite migration/fix mode affects how existing installations recover and upgrade.

Stored data model
Persistent data-model change detected: database schema: docs/refactor/database-first.md, database schema: src/state/openclaw-agent-db.test.ts, database schema: src/state/openclaw-agent-db.ts, database schema: src/state/openclaw-agent-schema.generated.ts, database schema: src/state/openclaw-agent-schema.sql, migration/backfill/repair: docs/channels/channel-routing.md, and 38 more. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #88838
Summary: this PR is the active implementation candidate for the canonical SQLite session/transcript migration tracker; older 3.2 foundation PRs are explicitly superseded and PR #98090 is an adjacent overflow fix that references this lane.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Restore explicit-file overflow truncation for non-SQLite sessionFile targets.
  • Resolve the dirty merge state and refresh focused validation on the merge result.
  • Get maintainer acceptance for the plugin runtime helper removal and forward-only SQLite migration behavior.

Mantis proof suggestion
A real Telegram conversation across the SQLite flip would demonstrate user-visible session continuity better than local row-count proof alone. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis telegram desktop proof: exercise an existing Telegram conversation across the SQLite session/transcript flip, send a follow-up message, and show the reply stays attached to the expected session with no stale JSONL dependency; redact private chat/user details.

Risk before merge

  • [P1] Explicit session-file runs can keep oversized tool results after compaction because the new runtime-transcript truncation helper rejects non-SQLite targets instead of using the existing file-backed truncation path.
  • [P1] The PR changes the persistent per-agent session/transcript data model and active runtime store, so merge requires maintainer acceptance of forward-only migration, archive-before-delete behavior, and upgrade proof.
  • [P1] The PR removes deprecated plugin runtime session helper methods, which is an intentional compatibility-sensitive SDK/runtime contract cutover but still needs maintainer ownership before landing.
  • [P1] The PR is currently dirty/conflicting against main, so the reviewed head needs a refreshed merge result and validation before it can be treated as land-ready.

Maintainer options:

  1. Repair Explicit-File Overflow Recovery (recommended)
    Preserve the current file-backed truncation branch for explicit non-SQLite sessionFile targets before rerunning focused overflow and migration validation.
  2. Accept The Storage/API Cutover
    Maintainers may intentionally accept the forward-only SQLite migration and deprecated runtime helper removals after reviewing the refreshed upgrade proof and docs.
  3. Pause Or Split The Migration
    If the dirty merge state or API/storage cutover is too broad to review safely, pause this branch and split the remaining migration, SDK cleanup, or doctor work into narrower PRs.
Copy recommended automerge instruction
@clawsweeper automerge

Special instructions:
Preserve file-backed explicit sessionFile overflow truncation when the runtime transcript target is not SQLite-backed, then rerun the focused overflow-compaction tests and migration proof commands relevant to this PR.

Next step before merge

  • [P2] Protected labels, the dirty merge state, and storage/API compatibility decisions require maintainer handling even though the explicit truncation repair itself is mechanical.

Security
Cleared: The diff is security-sensitive because it migrates local session/transcript state and touches plugin/runtime surfaces, but no concrete supply-chain or security-boundary regression was found in the inspected files.

Review findings

  • [P2] Preserve explicit session-file truncation fallback — src/agents/embedded-agent-runner/run.ts:2993-2999
Review details

Best possible solution:

Repair explicit-file overflow truncation, resolve the merge conflict, and land only after maintainers accept the SQLite storage/API/doctor cutover with fresh upgrade proof.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection: explicit sessionFile targets remain valid, but the PR run path now invokes a SQLite-only runtime transcript truncation helper that returns without truncating non-SQLite targets; no tests were run because this was a read-only review.

Is this the best way to solve the issue?

No as-is: the storage-flip direction matches the tracker, but the patch needs the explicit-file truncation repair plus refreshed merge and upgrade proof before it is the best landing shape.

Full review comments:

  • [P2] Preserve explicit session-file truncation fallback — src/agents/embedded-agent-runner/run.ts:2993-2999
    Both post-compaction and overflow recovery now call truncateOversizedToolResultsInRuntimeTranscript, but explicit sessionFile run targets remain valid and can resolve to a non-SQLite file artifact. The runtime helper rejects non-SQLite targets and returns a non-truncating result, so explicit-file runs can keep oversized tool results that current main truncated; branch to truncateOversizedToolResultsInSession for non-SQLite targets.
    Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 786abe78df53.

Label changes

Label justifications:

  • P1: The PR changes core session/transcript runtime storage and message continuity for real agent/channel workflows.
  • merge-risk: 🚨 compatibility: The branch removes deprecated plugin runtime helpers and introduces a forward-only SQLite migration/doctor path for existing state.
  • merge-risk: 🚨 session-state: The branch moves active sessions and hot transcripts into SQLite, where migration or target-resolution mistakes can stale or mis-associate agent state.
  • merge-risk: 🚨 message-delivery: Session/transcript continuity affects whether channel follow-ups attach to the intended conversation and preserve reply history.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body and follow-up comments include after-change live output for migration/import validation, direct SQLite row checks, and focused proof commands for the storage flip.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body and follow-up comments include after-change live output for migration/import validation, direct SQLite row checks, and focused proof commands for the storage flip.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The PR changes session/transcript continuity that is visible in Telegram follow-up replies, so a short Telegram Desktop proof would materially help review.
Evidence reviewed

Acceptance criteria:

  • [P1] Read-only review only; no tests were run.
  • [P1] After repair, run focused overflow-compaction tests covering explicit file-backed sessionFile targets and SQLite-backed runtime transcript targets.
  • [P1] After conflict resolution, rerun the PR's migration/import/doctor proof on the actual merge result.
  • [P1] For Telegram-visible confidence, capture a Telegram Desktop proof of a follow-up reply staying on the expected session after migration.

What I checked:

Likely related people:

  • jalehman: Authored the canonical tracker and active flip PR and appears repeatedly in recent session accessor, SQLite migration, plugin runtime, and Codex transcript mirror history. (role: feature owner and recent area contributor; confidence: high; commits: 258b83c; files: src/config/sessions/session-accessor.ts, src/plugins/runtime/types-core.ts, extensions/codex/src/app-server/transcript-mirror.ts)
  • vincentkoc: Recent history ties this author to tool-result projection and truncation paths affected by the overflow-truncation regression. (role: adjacent owner; confidence: medium; files: src/agents/embedded-agent-runner/tool-result-truncation.ts)
  • steipete: Prior database-first runtime state and plugin SDK work overlaps the stored-state and plugin-runtime compatibility surface changed by this PR. (role: adjacent owner; confidence: medium; files: src/state/openclaw-agent-schema.sql, src/plugins/runtime/types-core.ts)
  • yungchentang: Recent session-accessor history includes archive and migration-adjacent work relevant to the archive-before-delete boundary. (role: adjacent contributor; confidence: low; files: src/config/sessions/session-accessor.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 25, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the scripts Repository scripts label Jun 25, 2026
@jalehman

Copy link
Copy Markdown
Contributor Author

Behavior proof for the storage flip follow-up at 7d4e7d5af0223ada5bdccc1e0d31d68399c2116f:

  • Addressed the ClawSweeper SDK storage finding: readSessionTranscriptEvents, readLatestAssistantTextByIdentity, assistant mirror appends, write locks, and transcript update publication now use the SQLite accessor path for scoped transcript identities instead of reading/writing the legacy JSONL locator. The only file write fallback left is the explicit-file/no-session-key legacy path.
  • Restored the public resolveAndPersistSessionFile SDK export and updated the accessor-boundary allowlist/test so this remains intentional.
  • Preserved mirror append contracts: updateMode: "none" does not publish, idempotency replays do not republish, unkeyed delivery mirrors still dedupe by latest equivalent assistant text, and stale session bindings return session-rebound.

Live local migration proof against Josh's local OpenClaw store:

  • Original pre-import backup exists at /tmp/openclaw-session-sqlite-preimport-20260625T030716Z.tar.gz.
  • A later read-only validate found live drift after the first import, so I made a targeted row backup before re-importing: /tmp/openclaw-main-session-sqlite-targeted-20260625T040207Z.json (about 3.2 MB).
  • Re-ran runDoctorSessionSqlite({ mode: "import", allAgents: true }): imported 58 legacy entries and 4547 transcript events. The only import-time issues were two pre-existing missing legacy JSONL transcript files.
  • Re-ran runDoctorSessionSqlite({ mode: "validate", allAgents: true }): issues: 0, validatedEntries: 58, validatedTranscriptEvents: 4547, sqliteEntries: 315, targets: 47.
  • Direct SQLite probe after validate:
    • main: 311 session entries, 36696 transcript events.
    • codex-proof: 2 session entries, 15 transcript events.
    • codex-image-proof: 2 session entries, 18 transcript events.

Local verification:

  • node scripts/run-vitest.mjs src/plugin-sdk/session-transcript-runtime.test.ts -> 17 tests passed.
  • node scripts/run-vitest.mjs src/plugin-sdk/session-transcript-runtime.test.ts src/config/sessions/session-accessor.conformance.test.ts src/config/sessions/session-accessor.test.ts -> 74 tests passed across 2 shards.
  • node scripts/run-tsgo.mjs -p tsconfig.core.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core.tsbuildinfo -> passed.
  • node scripts/check-session-accessor-boundary.mjs -> passed.
  • git diff --check -> passed.
  • .agents/skills/autoreview/scripts/autoreview --mode local --base upstream/main -> clean, no accepted/actionable findings.

Thermonuclear review note: the behavior fixes are materially cleaner now, but the strict maintainability review still flags the large SQLite backend/test shape from the broader branch (session-accessor.sqlite.ts is over 2.5k LOC and the conformance test is over 1.5k LOC). I did not split that architecture in this follow-up because the active blocker was the SDK storage contract regression and the branch already had live migration proof.

@jalehman

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 25, 2026
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime security Security documentation docker Docker and sandbox tooling labels Jun 25, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 25, 2026
@jalehman

Copy link
Copy Markdown
Contributor Author

Superseded by #98236, which carries the same SQLite sessions/transcripts storage flip on canonical upstream branch sqlite/sessions-transcripts-storage-flip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui channel: discord Channel integration: discord channel: feishu Channel integration: feishu channel: matrix Channel integration: matrix channel: mattermost Channel integration: mattermost channel: slack Channel integration: slack channel: telegram Channel integration: telegram channel: whatsapp-web Channel integration: whatsapp-web cli CLI command changes commands Command implementations docker Docker and sandbox tooling docs Improvements or additions to documentation extensions: acpx extensions: codex extensions: copilot extensions: memory-core Extension: memory-core extensions: ollama extensions: qa-lab gateway Gateway runtime maintainer Maintainer-authored PR mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. plugin: file-transfer proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. scripts Repository scripts security Security documentation size: XL status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant