Skip to content

refactor: add SQLite embedded run target adapter#90438

Closed
jalehman wants to merge 30 commits into
clawdbot-587/32-sqlite-store-foundationfrom
clawdbot-d02.1.9.1.29/32-embedded-run-session-target-adapter
Closed

refactor: add SQLite embedded run target adapter#90438
jalehman wants to merge 30 commits into
clawdbot-587/32-sqlite-store-foundationfrom
clawdbot-d02.1.9.1.29/32-embedded-run-session-target-adapter

Conversation

@jalehman

@jalehman jalehman commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

What

Implements the SQLite adapter side of the embedded-agent run/session target seam. The adapter resolves storage-neutral run/session targets through SQLite-backed session access while retaining the named active artifact boundary needed by current embedded-run internals.

Why

This is the 3.2 counterpart to the file-backed embedded run/session target seam. It proves the target contract can operate under SQLite wiring before the production storage flip.

Changes

  • Adds SQLite target resolution
  • Persists active artifact identity
  • Handles stale artifact recovery
  • Covers consult fork scoping

Testing

  • node scripts/run-vitest.mjs src/agents/run-session-target.test.ts src/context-engine/context-engine.test.ts src/commitments/runtime.test.ts src/crestodian/assistant.test.ts src/hooks/llm-slug-generator.test.ts src/talk/agent-consult-runtime.test.ts src/agents/cli-runner/prepare.test.ts src/agents/cli-runner.before-agent-reply-cron.test.ts src/agents/embedded-agent-runner/run.overflow-compaction.test.ts
  • node scripts/run-vitest.mjs src/config/sessions/session-accessor.conformance.test.ts src/config/sessions/session-accessor.test.ts src/state/openclaw-agent-db.test.ts src/plugin-sdk/session-transcript-hit.test.ts
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base clawdbot-587/32-sqlite-store-foundation --no-web-search --thinking codex=low

Refs #88838

jalehman added 29 commits May 31, 2026 17:28
@openclaw-barnacle openclaw-barnacle Bot added the agents Agent runtime and tooling label Jun 4, 2026
@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 24, 2026, 7:28 PM ET / 23:28 UTC.

Summary
The PR adds a SQLite-backed embedded run/session target adapter and related accessor, schema, runtime, and test changes for the Path 3 session/transcript SQLite migration stack.

PR surface: Source +2956, Tests +2164, Generated +220. Total +5340 across 44 files.

Reproducibility: yes. Source inspection of PR head shows the explicit compaction file target being dropped when a session key exists, SQLite reads using a schema-creating opener, and session cleanup deleting rows before an enforced export receipt.

Review metrics: 2 noteworthy metrics.

  • Run Target Contract Changes: 3 runtime parameter contracts changed. Making sessionFile optional and adding storage-neutral sessionTarget changes how embedded, CLI, and compaction callers identify session state.
  • Agent SQLite Schema Change: schema version 1 -> 2; 7 session/transcript tables added. The per-agent persisted schema change makes upgrade, preview, retention, and migration behavior important before merge.

Stored data model
Persistent data-model change detected: database schema: src/agents/embedded-agent-runner/run.overflow-compaction.test.ts, database schema: src/state/openclaw-agent-db.test.ts, database schema: src/state/openclaw-agent-schema.generated.ts, database schema: src/state/openclaw-agent-schema.sql, serialized state: src/agents/embedded-agent-runner/run.overflow-compaction.test.ts, serialized state: src/auto-reply/reply/session-fork.ts, and 21 more. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #88838
Summary: This PR is an implementation slice for the canonical session/transcript SQLite migration tracker and depends on the open SQLite foundation while building on merged file-backed seam work.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Preserve explicit compaction session files and add focused regression coverage.
  • [P1] Add no-create SQLite read paths and enforce transcript export receipts before cascading deletes.
  • [P1] Rebase the draft stack and add redacted real runtime terminal/log/live-output proof for the SQLite run-target path.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body lists focused tests and autoreview only; it still needs redacted terminal, log, live-output, recording, or linked-artifact proof from a real SQLite run-target path before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The branch is draft and GitHub reports it as conflicting/dirty, so the current head cannot merge without a rebase and refreshed review.
  • [P1] The PR changes runtime session identity and persisted per-agent SQLite session/transcript behavior; upgrade, preview, retention, and deletion semantics need maintainer-visible proof before merge.
  • [P1] The PR body lists focused tests and autoreview only, not redacted terminal/log/live-output/recording proof for a real SQLite run-target path.

Maintainer options:

  1. Fix Storage-Target Blockers First (recommended)
    Preserve explicit compaction session files, add no-create/read-only SQLite read paths, require transcript export receipts before deletes, rebase the stack, and refresh proof before merge.
  2. Accept Only As Gated Tracker Work
    Maintainers could keep the slice open only if the broader tracker continues to block the runtime flip on preview, retention, migration, and proof requirements.
  3. Pause Until Foundation Settles
    Because the PR is stacked on the SQLite foundation, maintainers can wait for the foundation PR and current seam contract to stabilize before refreshing this draft.

Next step before merge

  • [P1] Protected draft storage-migration work has storage-safety findings, a dirty base, and missing real proof, so the next action is author/maintainer handling rather than an automated repair lane.

Security
Needs attention: No new dependency or third-party execution path was found, but SQLite transcript retention/deletion behavior is security-sensitive and needs attention.

Review findings

  • [P1] Preserve explicit compact session files — src/context-engine/delegate.ts:67
  • [P1] Use no-create opens for SQLite read helpers — src/config/sessions/session-accessor.sqlite.ts:104
  • [P1] Require export receipts before transcript deletes — src/config/sessions/session-accessor.sqlite.ts:751
Review details

Best possible solution:

Keep the PR open under the tracker, rebase it onto the current stack, fix the storage-safety blockers, and add redacted real SQLite run-target proof before any merge decision.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection of PR head shows the explicit compaction file target being dropped when a session key exists, SQLite reads using a schema-creating opener, and session cleanup deleting rows before an enforced export receipt.

Is this the best way to solve the issue?

No. The SQLite adapter seam is the intended owner boundary, but the current draft is not the best mergeable shape until it preserves current target behavior, adds no-create read paths, guards transcript deletion, rebases, and provides real proof.

Full review comments:

  • [P1] Preserve explicit compact session files — src/context-engine/delegate.ts:67
    delegateCompactionToRuntime now omits params.sessionFile whenever any session key is present. A ContextEngine caller that supplies sessionKey plus an explicit sessionFile but no fully resolved runtime target will be resolved through the configured/default store instead of compacting the caller's transcript file.
    Confidence: 0.9
  • [P1] Use no-create opens for SQLite read helpers — src/config/sessions/session-accessor.sqlite.ts:104
    The SQLite read helpers call the normal agent database opener, which creates directories/files, applies schema DDL, caches the handle, and registers the DB. Doctor/startup preview reads need a no-create/read-only path that returns empty results when the database is absent.
    Confidence: 0.9
  • [P1] Require export receipts before transcript deletes — src/config/sessions/session-accessor.sqlite.ts:751
    Deleting a sessions row cascades transcript_events, but lifecycle cleanup only counts transcript state as archived and does not require a durable export receipt first. Once SQLite transcripts are live, this can silently drop conversation history.
    Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3ab8d6aa609a.

Label changes

Label justifications:

  • P2: This is normal-priority storage migration work with significant future blast radius but no immediate production storage flip in the draft PR.
  • merge-risk: 🚨 compatibility: The PR changes embedded/CLI/compaction runtime session parameter contracts and SQLite open/schema behavior that can affect existing callers and upgrades.
  • merge-risk: 🚨 session-state: The diff rewires session identity, compaction target resolution, and transcript/session persistence paths.
  • merge-risk: 🚨 security-boundary: SQLite transcript rows can contain sensitive conversation history, and lifecycle deletion/export ordering affects the privacy and retention boundary.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists focused tests and autoreview only; it still needs redacted terminal, log, live-output, recording, or linked-artifact proof from a real SQLite run-target path before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +2956, Tests +2164, Generated +220. Total +5340 across 44 files.

View PR surface stats
Area Files Added Removed Net
Source 28 3065 109 +2956
Tests 13 2175 11 +2164
Docs 0 0 0 0
Config 0 0 0 0
Generated 3 222 2 +220
Other 0 0 0 0
Total 44 5462 122 +5340

Security concerns:

  • [medium] Sensitive transcript retention guard missing — src/config/sessions/session-accessor.sqlite.ts:751
    The SQLite cleanup path can delete session rows under cascading transcript foreign keys before a durable transcript export receipt is enforced, risking loss of sensitive conversation history without the required retention artifact.
    Confidence: 0.86

What I checked:

  • Repository policy applied: Root and scoped AGENTS.md files were read from disk; storage/session compatibility, persistent data migration, plugin-SDK boundary, and proof rules apply to this PR. (AGENTS.md:30, 3ab8d6aa609a)
  • Live PR state: GitHub reports this PR open, draft, conflicting/dirty, head 1c59b42, with maintainer, proof-needed, and storage/session/security merge-risk labels. (1c59b42c0362)
  • Current-main seam state: Current main has the file-backed run session target seam, but not the SQLite runtime target branch added by this PR. (src/agents/run-session-target.ts:20, 3ab8d6aa609a)
  • Explicit compaction target regression: PR head drops runtimeContext.sessionFile and only forwards params.sessionFile when no session key exists, changing explicit compaction-file targeting. (src/context-engine/delegate.ts:67, 1c59b42c0362)
  • Current-main compaction comparison: Current main unconditionally forwards params.sessionFile into compactEmbeddedAgentSessionDirect, so the PR changes existing compaction target behavior. (src/context-engine/delegate.ts:58, 3ab8d6aa609a)
  • SQLite reads use creating opener: The PR's SQLite read helpers call openOpenClawAgentDatabase, whose opener creates directories, opens the SQLite file, ensures schema, caches/registers the DB, and applies permissions. (src/config/sessions/session-accessor.sqlite.ts:104, 1c59b42c0362)

Likely related people:

  • jalehman: Authored the canonical tracker, this PR head, the merged file-backed embedded run target seam, the merged session accessor seam, and current-main adjacent session metadata accessor work. (role: stack coordinator and recent area contributor; confidence: high; commits: 1c59b42c0362, 0dfa22c6e0fc, ef47dd610c87; files: src/agents/run-session-target.ts, src/config/sessions/session-accessor.ts, src/context-engine/delegate.ts)
  • Shakker: Current-main blame for the central run-session-target and delegate files points at a broad recent commit that carried this area forward after the stack moved. (role: recent current-main committer; confidence: medium; commits: 8ecbf83c678c; files: src/agents/run-session-target.ts, src/context-engine/delegate.ts)
  • Vincent Koc: Committed the revert that deferred the earlier SQLite session metadata migration, which the tracker cites as lessons for this staged reland. (role: adjacent storage migration contributor; confidence: medium; commits: 7f1d82ab2518; files: src/state/openclaw-agent-db.ts, src/config/sessions/store.ts)
  • Peter Steinberger: History for the context-engine delegate includes earlier runtime import and compaction-boundary maintenance that is adjacent to this PR's delegate changes. (role: adjacent context-engine contributor; confidence: low; commits: 87fa88ac3d35; files: src/context-engine/delegate.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 4, 2026
@jalehman
jalehman force-pushed the clawdbot-587/32-sqlite-store-foundation branch 2 times, most recently from 1ccad0c to 07e48cf Compare June 11, 2026 05:04
@jalehman
jalehman force-pushed the clawdbot-587/32-sqlite-store-foundation branch from 07e48cf to ba3d281 Compare June 14, 2026 14:21
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jun 16, 2026
@clawsweeper clawsweeper Bot removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 16, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jun 19, 2026
@jalehman
jalehman force-pushed the clawdbot-587/32-sqlite-store-foundation branch from 543fb5c to ffd2ae9 Compare June 24, 2026 23:10
@jalehman

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #96625. The SQLite storage flip PR now carries the remaining Path 3 implementation line, including the foundation/adapter work that this older 3.2 PR was meant to stage. Please use #96625 and tracker #88838 for remaining review/proof instead of rebasing this stale split PR.

@jalehman jalehman closed this Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant