refactor: move runtime state to SQLite#81402
Conversation
github-actions
Bot
added
the
dependencies-changed
Dependency Changes DetectedThis PR changes dependency-related files. Maintainers should confirm these changes are intentional. Changed files:
Maintainer follow-up:
|
openclaw-barnacle
Bot
added
docs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0694638fe9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const config = fs.existsSync(configPath) ? readJson(configPath) : {}; | ||
| const allowLegacyCompat = process.env.OPENCLAW_PACKAGE_ACCEPTANCE_LEGACY_COMPAT === "1"; | ||
| if (!allowLegacyCompat && !index.installRecords) { | ||
| const index = readInstalledPluginIndex(); |
There was a problem hiding this comment.
In this package-acceptance assertions script, the new SQLite-backed reads call readInstalledPluginIndex, writeInstalledPluginIndex, and readInstalledPluginRecords, but this file never imports or defines those helpers (unlike the sibling probes that import ../installed-plugin-index.mjs). Any command path that reaches install-record checks now throws ReferenceError before validating plugin install/uninstall behavior.
Useful? React with 👍 / 👎.
| const storePath = path.join(sessionsDir, "sessions.json"); | ||
| const store = readJson(storePath); | ||
| const entry = Object.values(store).find((candidate) => candidate?.sessionId === sessionId); | ||
| const entry = readAgentSessionEntryBySessionId(sessionId); |
There was a problem hiding this comment.
This Codex npm live assertion now calls readAgentSessionEntryBySessionId (and later countAgentTranscriptEvents / readPluginStateJson), but none of those helpers are imported or defined in this script or its existing codex-install-utils.mjs import. The assert-agent-turn phase therefore fails with ReferenceError after the agent turn succeeds, so the live Codex npm plugin lane can no longer complete.
Useful? React with 👍 / 👎.
| const authRaw = fs.readFileSync(authPath, "utf8"); | ||
| if (!authRaw.includes("OPENAI_API_KEY")) { | ||
| const authAgentDir = path.join(stateDir(), "agents", "main", "agent"); | ||
| const authStore = readAuthProfileStorePayload(authAgentDir); |
There was a problem hiding this comment.
The Codex on-demand assertion was switched to read the auth profile from SQLite, but readAuthProfileStorePayload is not defined or imported in this file. When this assertion script runs, it reaches this line after install/model checks and aborts with ReferenceError instead of validating the persisted auth profile.
Useful? React with 👍 / 👎.
| const index = fs.existsSync(indexPath) ? readJson(indexPath) : {}; | ||
| const cfg = fs.existsSync(configPath()) ? readJson(configPath()) : {}; | ||
| return index.installRecords || index.records || cfg.plugins?.installs || {}; | ||
| return withSqliteDatabase(stateDatabasePath(), (db) => { |
There was a problem hiding this comment.
The live plugin tool assertion now depends on withSqliteDatabase and stateDatabasePath here (and readMainAgentTranscript later), but this script does not import or define any of them. As a result, both install-record lookup and transcript validation paths fail immediately with ReferenceError, breaking the live plugin tool E2E lane.
Useful? React with 👍 / 👎.
| assertOpenAiRequestLogUsed, | ||
| } from "../agent-turn-output.mjs"; | ||
| import { applyMockOpenAiModelConfig } from "../fixtures/mock-openai-config.mjs"; | ||
| import { DatabaseSync } from "node:sqlite"; |
There was a problem hiding this comment.
This import block dropped assertAgentReplyContainsMarker and assertOpenAiRequestLogUsed, but assertAgentTurn still calls both helpers later in the same file. The npm onboard channel-agent scenario will therefore fail with ReferenceError during its agent-turn verification instead of checking the reply marker and mock OpenAI request log.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11715d136b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| struct CronSchedulerStatus: Decodable { | ||
| let enabled: Bool | ||
| let storePath: String | ||
| let storeKey: String |
There was a problem hiding this comment.
When the gateway is current, cron.status still returns storePath from src/cron/service/ops.ts:249-251 / CronStatusSummary, not storeKey. This required storeKey field makes cronStatus() fail decoding; CronJobsStore.refreshJobs catches that with try?, so the macOS Cron UI silently loses scheduler enabled/next-wake status whenever it talks to the existing gateway. Either keep decoding storePath here or add a compatibility decoder while the wire response is unchanged.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93dfb037aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| public struct OpenClawChatSessionsListResponse: Codable, Sendable { | ||
| public let ts: Double? | ||
| public let path: String? | ||
| public let databasePath: String? |
There was a problem hiding this comment.
When MacGatewayChatTransport.listSessions decodes the live gateway response, the gateway still sends path for sessions.list (SessionsListResultBase and listSessionsFromStoreAsync both use path), but this renamed Codable model has no CodingKeys fallback from path to databasePath. As a result decoded.databasePath is always nil for current gateways before WebChatSwiftUI forwards it, unlike the nearby macOS session/health decoders that preserve the old wire key.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e0c82f463
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fi | ||
| MIN_NODE_VERSION="22.19.0" | ||
| APK_NODE_BIN_DIR="/usr/bin" | ||
| SHARP_IGNORE_GLOBAL_LIBVIPS="${SHARP_IGNORE_GLOBAL_LIBVIPS:-1}" |
There was a problem hiding this comment.
When SHARP_IGNORE_GLOBAL_LIBVIPS is not already present in the environment, this assignment creates only a shell-local variable, and the later env ... npm install invocations do not inherit it. On hosts with a system libvips that sharp would otherwise try to use, the installer still behaves as if the default override were absent; export this value or pass it in the env command that launches npm.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9ae4443956
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| const authRaw = fs.readFileSync(authPath, "utf8"); | ||
| if (!authRaw.includes("OPENAI_API_KEY")) { | ||
| const authStore = readAuthProfileStorePayload(stateDir, agentDir); |
There was a problem hiding this comment.
In the npm onboard Docker lane (scripts/e2e/npm-onboard-channel-agent-docker.sh:160), this runs right after onboarding, but the current auth-profile writer is still file-backed: updateAuthProfileStoreWithLock creates auth-profiles.json and saveAuthProfileStore writes that path (src/agents/auth-profiles/store.ts:752-763, src/agents/auth-profiles/store.ts:1086-1102). Since nothing writes an auth_profile_stores row for this path, this SQLite lookup returns undefined and the subsequent legacy-file rejection fails even when onboarding persisted the expected env ref.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d08909f06d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,26 @@ | |||
| export { | |||
There was a problem hiding this comment.
This adds a src/plugin-sdk/provider-ai.ts facade, but neither package.json exports nor scripts/lib/plugin-sdk-entrypoints.json include provider-ai (and the sibling new provider-ai-oauth/sqlite-runtime surfaces are omitted the same way). In a published package, an external or bundled plugin importing openclaw/plugin-sdk/provider-ai will hit ERR_PACKAGE_PATH_NOT_EXPORTED even though the facade exists in source, so the SDK entrypoint list/package exports need to be updated with the new subpaths.
Useful? React with 👍 / 👎.
|
Updated this PR for the YAML-only QA scenario format in commit Change made:
I was not able to make a normal merge commit from current |
|
Closing this broad branch in favor of the incremental database-first work now on The branch is 333 files (+9,503/-4,514) and currently 7,775 commits behind The supported path is to keep landing any remaining storage gaps as narrow owner-boundary changes with explicit doctor migration and focused upgrade proof. A concrete missing shared seam that cannot be delivered incrementally would be reason to revisit; rebasing this 333-file snapshot is not. Preserving the branch for history, but closing the PR as superseded by the incremental architecture on |
4 participants
Summary
This PR reopens the database-first runtime-state refactor after the accidental direct landing in #78595 was reverted from
main. It moves active OpenClaw runtime state away from scattered JSON, JSONL, lock files, cache files, and sidecar stores into a typed SQLite layout, while keeping user configuration file-backed.The main contract is intentionally simple:
~/.openclaw/state/openclaw.sqlite.~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite.openclaw.json, plugin manifests, external provider credential files, and user workspaces do not move into SQLite.{ agentId, sessionId }plus explicit conversation/session metadata. Runtime code must not pass transcript file paths or pseudo-locators such assqlite-transcript://....What changed
openclaw doctor --fixstyle migration surfaces instead of runtime startup compatibility paths.mainfor this cleanup PR.Architecture
Global database:
state/openclaw.sqlitestate_leasestable used by SQLite-backed lock helpers.Agent database:
agents/<agentId>/agent/openclaw-agent.sqliteagent_databases.Proxy capture database:
src/proxy-capture/schema.sqlopenclaw.sqlite.Schema files
src/state/openclaw-state-schema.sql: global control-plane schema, 62 tables.src/state/openclaw-state-schema.generated.ts: generated SQL module for the global schema.src/state/openclaw-state-db.generated.d.ts: generated Kysely types for the global schema.src/state/openclaw-agent-schema.sql: per-agent data-plane schema, 19 tables.src/state/openclaw-agent-schema.generated.ts: generated SQL module for the per-agent schema.src/state/openclaw-agent-db.generated.d.ts: generated Kysely types for the per-agent schema.src/proxy-capture/schema.sql: standalone proxy capture schema, 3 tables.src/proxy-capture/schema.generated.tsandsrc/proxy-capture/db.generated.d.ts: generated SQL/types for proxy capture.Across these schema files there are 84 tables and 110 explicit indexes/unique indexes.
Global database schema
Core metadata and locking:
schema_meta: database role, schema version, app version, timestamps.agent_databases: registry of per-agent database paths, schema versions, last-seen timestamps, and size hints.state_leases: scoped durable lease/lock rows with owner, heartbeat, expiry, and payload.Auth, diagnostics, and approvals:
auth_profile_stores,auth_profile_state: runtime auth profile store/state JSON by store key.diagnostic_events,diagnostic_stability_bundles: scoped diagnostic payloads and generated stability bundles.exec_approvals_config: persisted exec approval configuration and derived counts.config_health_entries: last-known-good/promoted config health state.Pairing, device, node, and native app state:
device_pairing_pending,device_pairing_paired: device pairing requests and approved devices.device_bootstrap_tokens: bootstrap tokens plus issued/redeemed profile metadata.node_pairing_pending,node_pairing_paired: node pairing requests and approved nodes.device_identities,device_auth_tokens: device keypairs and role-scoped tokens.android_notification_recent_packages: Android notification recent-package ordering.macos_port_guardian_records: macOS port guardian records.workspace_setup_state: workspace setup lifecycle state.native_hook_relay_bridges: native hook relay bridge metadata and expiry.node_host_config: node-host config state.Models, media, push, and voice wake:
model_capability_cache: provider/model capability and cost cache.agent_model_catalogs: generated model catalogs per agent dir.managed_outgoing_image_records: managed outgoing image attachment metadata.web_push_subscriptions,web_push_vapid_keys: web push subscription and VAPID key state.apns_registrations: APNS registration rows for native nodes.voicewake_triggers,voicewake_routing_config,voicewake_routing_routes: voice wake trigger and routing state.Plugin and channel state:
installed_plugin_index: persisted installed-plugin index and diagnostics.plugin_state_entries: JSON plugin state by plugin, namespace, and key.plugin_blob_entries: binary plugin blobs by plugin, namespace, and key.channel_pairing_requests,channel_pairing_allow_entries: channel pairing codes and allow entries.plugin_binding_approvals: plugin/channel/account binding approvals.current_conversation_bindings: current channel conversation bindings to target agent/session rows.Gateway, replay, delivery, and runtime queues:
gateway_restart_sentinel,gateway_restart_intent,gateway_restart_handoff: restart continuity and supervisor handoff state.acp_replay_sessions,acp_replay_events: ACP replay session/event ledger.delivery_queue_entries: durable delivery queue with retry, failure, session, channel, target, and recovery metadata.command_log_entries: durable command log entries.tui_last_sessions: last active session per TUI scope.Tasks, flows, subagents, cron, and commitments:
task_runs: durable task run state.task_delivery_state: notification state per task.flow_runs: task-flow/controller state.subagent_runs: child-agent lifecycle, archive, announcement, frozen-result, and final-delivery state.cron_jobs: scheduler job definitions plus runtime state.cron_run_logs: scheduler run history.commitments: follow-up commitment records and delivery lifecycle.Storage, backup, migration, sandbox, and capture:
media_blobs: managed media blob storage.skill_uploads: temporary skill upload archives with idempotency/expiry.sandbox_registry_entries: sandbox/container registry state.migration_runs,migration_sources: doctor migration run/source ledger.backup_runs: backup archive manifest records.capture_sessions,capture_events,capture_blobs: managed proxy-capture session/event/blob storage.update_check_state: update notification/auto-install state.Per-agent database schema
Session and conversation identity:
schema_meta: agent database role, schema version, agent id, app version, timestamps.sessions: canonical session metadata, status, channel/account/model metadata, parent/spawn identity, and primary conversation reference.session_routes: maps stable session keys to canonical session ids.conversations: typed channel/account/conversation identity with direct/group/channel kind and thread/native ids.session_conversations: many-to-many session/conversation roles with one primary conversation per session.session_entries: compatibility/display entry JSON keyed by session key, linked to canonical session id.Transcripts and trajectory:
transcript_events: append-only transcript event stream keyed by(session_id, seq).transcript_event_identities: event id, parent/idempotency identity, tail indexes, and event type metadata.transcript_snapshots: explicit transcript snapshot records with reason/count/metadata.trajectory_runtime_events: runtime trajectory rows linked to session/run identity.acp_parent_stream_events: ACP parent stream event ledger by run id and sequence.Agent-local workspace, artifacts, memory, and cache:
vfs_entries: virtual filesystem namespace/path records with optional blob content and metadata.tool_artifacts: tool output artifacts by run id and artifact id.run_artifacts: run artifacts by run id and path.memory_index_meta: memory index provider/model/config metadata.memory_index_sources: indexed memory/session sources.memory_index_chunks: indexed text chunks with embedding blobs and source/session references.memory_embedding_cache: provider/model/provider-key/hash embedding cache.cache_entries: generic agent-local cache rows with JSON/blob payloads and optional expiry.Runtime contract changes reviewers should focus on
{ agentId, sessionId }or{ agentId, sessionKey }, not transcript file paths.sessions.session_scope,sessions.primary_conversation_id,conversations, andsession_conversationsare the typed routing source of truth.session_keyremains a routing/display key, not a parsing surface for hidden provider identity.transcript_eventsandtranscript_event_identities; JSONL transcript helpers should be limited to doctor import/export/debug materialization.sessions.json,.jsonl,.jsonl.lock, pruning/truncation files, auth profile files, model catalog files, and old sidecar stores should appear only in doctor migration, backup/import, or explicit legacy tests.Important non-goals
openclaw.jsoninto SQLite.1because this layout has not shipped yet.main.Real behavior proof
main.OPENCLAW_HOMEso runtime state is isolated from the maintainer profile.OPENCLAW_HOME=$(mktemp -d) pnpm openclaw doctorVerification
pnpm check:database-first-legacy-storespnpm db:kysely:checkpnpm lint:kyselygit diff --check && node scripts/check-no-conflict-markers.mjsReview notes
Revert "refactor: move runtime state to SQLite").