Persist plugin install index in SQLite by steipete · Pull Request #88794 · openclaw/openclaw

steipete · 2026-05-31T21:31:58Z

Summary

Persist the installed plugin index in the shared SQLite state database and remove steady-state JSON index writes/fallback reads.
Import legacy plugins/installs.json through doctor/preflight state migrations, including record-only legacy shapes, embedded plugin records, custom state dirs, and merge-with-existing SQLite rows.
Update E2E/docker/post-upgrade doctor paths to use or scrub the SQLite index, keep agent post-upgrade startup from missing legacy installed plugins, and honor disabled persisted-registry reads without touching SQLite.

Verification

pnpm test src/commands/doctor/shared/plugin-registry-migration.test.ts src/cli/argv.test.ts src/cli/program/config-guard.test.ts src/infra/state-migrations.state-dir.test.ts src/plugins/installed-plugin-index.test.ts src/plugins/installed-plugin-index-store.test.ts src/plugins/installed-plugin-index-records.test.ts src/plugins/plugin-registry.test.ts src/plugins/plugin-metadata-snapshot.memo.test.ts src/config/io.write-config.test.ts src/commands/doctor-state-migrations.test.ts test/scripts/live-docker-stage.test.ts test/scripts/docker-build-helper.test.ts src/plugins/manifest-registry-installed.test.ts src/plugins/manifest-metadata-scan.test.ts src/plugins/plugin-metadata-snapshot.memo.test.ts src/plugins/manifest-model-id-normalization.test.ts
pnpm test src/plugins/plugin-metadata-snapshot.memo.test.ts
pnpm tsgo:core
pnpm check:test-types
pnpm check:architecture
git diff --check origin/main...HEAD
/Users/steipete/Projects/agent-scripts/skills/autoreview/scripts/autoreview --mode branch --base origin/main clean after fixing two accepted findings

clawsweeper · 2026-05-31T21:33:55Z

Codex review: needs real behavior proof before merge. Reviewed May 31, 2026, 8:48 PM ET / 00:48 UTC.

Summary
The PR moves installed plugin index persistence from plugins/installs.json into the shared SQLite state DB and updates doctor, startup, E2E, and docs paths around that migration.

PR surface: Source +466, Tests +490, Docs 0, Other +181. Total +1137 across 56 files.

Reproducibility: yes. The blocking PR findings are source-reproducible from the live head: preferPersisted: false disables ledger reads, records-only legacy rows persist with empty plugins, and SQLite open errors are caught broadly.

Review metrics: 2 noteworthy metrics.

Durable state store changed: 1 JSON ledger retired; 1 SQLite table becomes canonical. Changing where plugin install metadata lives makes upgrade, downgrade, and doctor behavior important before merge.
Migration preflight scope changed: 3 command paths changed in argv expectations. models list, models status, and agent now consider state migration, which can affect startup behavior for existing users.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted terminal/log/live proof showing a real plugins/installs.json migration into SQLite with installed plugins visible afterward.
Preserve install-record reads when only persisted registry snapshots are bypassed.
Rebuild or mark stale records-only legacy rows and rethrow unsupported state DB errors.

Proof guidance:

[P1] Needs real behavior proof before merge: The PR body lists tests and checks only; it needs redacted terminal or log proof of a real legacy install-index migration with installed plugins visible afterward. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] The branch moves durable installed-plugin state into SQLite without redacted real upgrade proof from an actual legacy state directory.
[P1] preferPersisted: false can hide installed external plugin records from provider/model/auth metadata callers that only meant to bypass persisted registry snapshots.
[P1] Broad SQLite read catches can convert unsupported state database versions into missing plugin index behavior instead of failing closed.
[P1] Records-only legacy JSON can be persisted as an apparently usable index with an empty plugin list and no diagnostics.

Maintainer options:

Fix compatibility and proof before merge (recommended)
Preserve current install-record semantics, narrow SQLite read error handling, handle records-only legacy rows safely, and refresh redacted real migration proof on the exact head.
Accept stricter persisted-disable semantics
Maintainers could intentionally make preferPersisted: false disable all SQLite install-ledger reads, but that needs explicit docs and tests for provider/model/auth fallout.
Pause until upgrade evidence exists
Because this changes durable state, maintainers can pause the PR until the contributor shows legacy installs migrating into SQLite and remaining visible afterward.

Next step before merge

[P1] Human handling is needed because the PR has a protected maintainer label, missing contributor-side migration proof, and compatibility findings that should be fixed and reviewed on the exact head.

Security
Cleared: No concrete security or supply-chain regression was found; the blockers are compatibility and provider/auth metadata correctness.

Review findings

[P1] Keep install records when bypassing persisted registry — src/plugins/plugin-registry-snapshot.ts:448
[P2] Rebuild records-only legacy indexes before persisting — src/infra/state-migrations.ts:314-323
[P2] Do not hide unsupported state DB errors — src/plugins/installed-plugin-index-store.ts:323-324

Review details

Best possible solution:

Keep the SQLite migration, preserve install-ledger reads when only persisted snapshots are bypassed, rebuild or mark stale records-only legacy rows, rethrow state DB compatibility errors, and require redacted real migration proof before merge.

Do we have a high-confidence way to reproduce the issue?

Yes. The blocking PR findings are source-reproducible from the live head: preferPersisted: false disables ledger reads, records-only legacy rows persist with empty plugins, and SQLite open errors are caught broadly.

Is this the best way to solve the issue?

No. Moving the index to SQLite matches repository direction, but the current patch is not the best merge shape until it preserves existing registry semantics, handles legacy rows safely, and proves a real upgrade path.

Full review comments:

[P1] Keep install records when bypassing persisted registry — src/plugins/plugin-registry-snapshot.ts:448
At this line, preferPersisted: false also disables reading the install ledger, so the derived path below receives {} records. Provider-scoped discovery uses this option to bypass snapshots, not installed records, so installed external plugins can disappear from provider/model/auth metadata.
Confidence: 0.9
[P2] Rebuild records-only legacy indexes before persisting — src/infra/state-migrations.ts:314-323
Records-only legacy JSON is converted into a persisted index with legacy policy fields, no diagnostics, and an empty plugins array. No-config registry reads can accept that row as persisted, so migration should rebuild from the records or make the synthetic row unambiguously stale.
Confidence: 0.84
[P2] Do not hide unsupported state DB errors — src/plugins/installed-plugin-index-store.ts:323-324
This broad catch also catches openOpenClawStateDatabase compatibility errors such as a newer user_version. That makes an unsupported state DB look like a missing plugin index and can silently hide installed plugins instead of surfacing the hard compatibility error.
Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.9

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 82d24b26ea93.

Label changes

Label justifications:

P2: This is a normal-priority persistence migration with concrete compatibility blockers but limited blast radius before merge.
merge-risk: 🚨 compatibility: The PR changes durable plugin install state, migration behavior, fallback reads, and downgrade/error semantics for existing state directories.
merge-risk: 🚨 auth-provider: The install-record read regression can remove installed external provider/auth metadata from model discovery and SecretRef-related paths.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🧂 unranked krab.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists tests and checks only; it needs redacted terminal or log proof of a real legacy install-index migration with installed plugins visible afterward. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +466, Tests +490, Docs 0, Other +181. Total +1137 across 56 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	18	675	209	+466
Tests	22	779	289	+490
Docs	2	3	3	0
Config	0	0	0	0
Generated	0	0	0	0
Other	14	254	73	+181
Total	56	1711	574	+1137

What I checked:

Repository policy applied: Root and scoped plugin policy were read; plugin state belongs in SQLite, but plugin registry, config loading, migrations, provider routing, and fallback behavior are compatibility-sensitive review surfaces. (AGENTS.md:30, 82d24b26ea93)
Current main still uses JSON install index: Current main resolves the installed plugin index to plugins/installs.json, so the central SQLite persistence change is not already implemented on main. (src/plugins/installed-plugin-index-store-path.ts:4, 82d24b26ea93)
PR drops install-record reads with preferPersisted: false: On the live PR head, persistedInstallRecordReadsEnabled is tied to persistedReadsEnabled, so caller-level snapshot bypass also bypasses the install ledger. (src/plugins/plugin-registry-snapshot.ts:448, a67273881c5a)
Current main preserves install-record reads: Current main keeps installed-plugin record reads enabled unless the environment break-glass disables persisted registry reads, which preserves provider/model/auth metadata for installed external plugins. (src/plugins/plugin-registry-snapshot.ts:444, 82d24b26ea93)
Legacy records-only migration can persist an empty plugin list: The PR synthesizes a legacy index with plugins: [] and no diagnostics for records-only JSON, which can be accepted as a persisted registry before discovery rebuilds the plugin records. (src/infra/state-migrations.ts:314, a67273881c5a)
Unsupported state DB errors are currently hard failures: The shared state DB opener throws when the database user_version is newer than this build supports; the PR's installed-index read catch can turn that into a missing-index result. (src/state/openclaw-state-db.ts:83, 82d24b26ea93)

Likely related people:

vincentkoc: Current main blame for the plugin registry snapshot, installed-index path/store, and state migration files points to recent work under Vincent Koc on the central surfaces this PR changes. (role: recent area contributor; confidence: medium; commits: 7c315252d68b, 015c6b40aeac; files: src/plugins/plugin-registry-snapshot.ts, src/plugins/installed-plugin-index-store-path.ts, src/plugins/installed-plugin-index-store.ts)
steipete: The live PR branch is authored by steipete and he also appears in recent current-main history, so he is a likely follow-up owner for branch repair and proof even though current-main blame for the central plugin files is shared. (role: PR branch owner and adjacent contributor; confidence: medium; commits: a7e7bcaef17d, cc4375fa62ee, e93216080aa1; files: src/plugins/installed-plugin-index-store.ts, src/plugins/plugin-registry-snapshot.ts, src/infra/state-migrations.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: abda602ca1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-31T23:12:14Z

  const disabledByEnv = hasEnvFlag(env, DISABLE_PERSISTED_PLUGIN_REGISTRY_ENV);
  const persistedReadsEnabled = !disabledByCaller && !disabledByEnv;
-  const persistedInstallRecordReadsEnabled = !disabledByEnv;
+  const persistedInstallRecordReadsEnabled = persistedReadsEnabled;


Keep install records when bypassing persisted registry

When callers set preferPersisted: false (for example provider-scoped discovery and provider env-var metadata), this line now also disables reading the install ledger, so the derived path below passes installRecords: {} instead of allowing loadInstalledPluginIndexWithDiscovery to load the SQLite install records. Installed external plugins that are only reachable through the ledger are therefore omitted from provider/model/auth metadata whenever the caller only wanted to bypass the persisted snapshot; before this change persisted metadata was skipped but install records were still read unless the env break-glass disabled them.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da9b029a61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-31T23:19:29Z

+    return parseInstalledPluginIndex({
+      version: INSTALLED_PLUGIN_INDEX_VERSION,
+      hostContractVersion: "legacy",
+      compatRegistryVersion: "legacy",
+      migrationVersion: INSTALLED_PLUGIN_INDEX_MIGRATION_VERSION,
+      policyHash: "legacy",
+      generatedAtMs: 0,
+      installRecords,
+      plugins: [],
+      diagnostics: [],


Rebuild records-only legacy indexes before persisting

When plugins/installs.json is a legacy records-only shape and there is no current SQLite row, this synthetic index is written to installed_plugin_index with hostContractVersion: "legacy" and an empty plugins array. loadPluginRegistrySnapshotWithMetadata({ stateDir }) only checks the policy hash when a config is supplied, so no-config registry callers can accept this row as source: "persisted" and see zero plugin records even though the migrated install records are present. Please persist a freshly derived index from the migrated records, or make these legacy envelopes unambiguously stale so the registry is rebuilt before being used.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f55198acf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-01T00:24:24Z

+  } catch {
+    return null;


Do not hide unsupported state DB errors

When the canonical SQLite state DB exists but cannot be opened with this build (for example after a downgrade, openOpenClawStateDatabase throws on a newer PRAGMA user_version in src/state/openclaw-state-db.ts), this broad catch turns that hard compatibility error into null. Registry callers then treat the install index as missing and can run with installed external plugins silently absent instead of stopping with the existing unsupported-schema error, so only malformed/missing index rows should be downgraded to null here.

Useful? React with 👍 / 👎.

# Conflicts: # src/commands/agent-command.test-mocks.ts

github-actions · 2026-06-01T00:47:08Z

Dependency Guard

This PR changes dependency-related files. Maintainers should confirm these changes are intentional.

Changed files:

pnpm-lock.yaml
ui/package.json

Maintainer follow-up:

Review whether the dependency changes are intentional.
Inspect resolved package deltas when lockfile, shrinkwrap, or workspace dependency policy changes are present.
Treat package-lock.json and npm-shrinkwrap.json diffs as security-review surfaces.
Run pnpm deps:changes:report -- --base-ref origin/main --markdown /tmp/dependency-changes.md --json /tmp/dependency-changes.json locally for detailed release-style evidence.

github-actions · 2026-06-01T00:47:19Z

Dependency graph change authorized

This PR includes dependency graph changes. A member of @openclaw/openclaw-secops authorized this exact head SHA with /allow-dependencies-change.

Approved SHA: 4d68180a3b563aac86acc6aedcd91f4420d806c5
Approved by: @steipete
Reason: remove unused ui three dependency after dependency-check failure

A later push changes the PR head SHA and requires a fresh security approval.

steipete · 2026-06-01T00:47:40Z

/allow-dependencies-change remove unused ui three dependency after dependency-check failure

* refactor: persist plugin install index in sqlite * fix: merge legacy plugin index records into sqlite * test: update plugin index sqlite fixtures * fix: migrate custom plugin install indexes * test: update plugin index sentinel * fix: exclude migrated plugin index archives * fix: read post-upgrade plugin index from sqlite * fix: migrate legacy plugin index before agent runs * fix: respect disabled persisted plugin registry reads * test: type plugin install record fixtures * fix: simplify plugin index record reader type * test: fix sqlite plugin index CI fallout * test: mock provider normalization in agent command tests # Conflicts: # src/commands/agent-command.test-mocks.ts * build: remove unused ui three dependency

steipete requested a review from a team as a code owner May 31, 2026 21:31

openclaw-barnacle Bot added docs Improvements or additions to documentation cli CLI command changes scripts Repository scripts commands Command implementations docker Docker and sandbox tooling size: XL maintainer Maintainer-authored PR labels May 31, 2026

steipete force-pushed the codex/sqlite-plugin-install-index branch from 180e020 to 354e907 Compare May 31, 2026 22:37

openclaw-barnacle Bot added the channel: feishu Channel integration: feishu label May 31, 2026

steipete force-pushed the codex/sqlite-plugin-install-index branch from 354e907 to abda602 Compare May 31, 2026 23:08

openclaw-barnacle Bot removed the channel: feishu Channel integration: feishu label May 31, 2026

chatgpt-codex-connector Bot reviewed May 31, 2026

View reviewed changes

steipete force-pushed the codex/sqlite-plugin-install-index branch from 9a523ba to 40c0efb Compare May 31, 2026 23:24

openclaw-barnacle Bot added the app: web-ui App: web-ui label May 31, 2026

steipete force-pushed the codex/sqlite-plugin-install-index branch from bb210b9 to 2697ae3 Compare May 31, 2026 23:33

openclaw-barnacle Bot added the agents Agent runtime and tooling label May 31, 2026

steipete force-pushed the codex/sqlite-plugin-install-index branch from fb55ee9 to 5666478 Compare May 31, 2026 23:53

openclaw-barnacle Bot removed app: web-ui App: web-ui agents Agent runtime and tooling labels May 31, 2026

clawsweeper Bot added the merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. label Jun 1, 2026

openclaw-barnacle Bot added agents Agent runtime and tooling extensions: github-copilot labels Jun 1, 2026

steipete force-pushed the codex/sqlite-plugin-install-index branch from b7c3b96 to 8d5c340 Compare June 1, 2026 00:16

openclaw-barnacle Bot removed the extensions: github-copilot label Jun 1, 2026

chatgpt-codex-connector Bot reviewed Jun 1, 2026

View reviewed changes

steipete force-pushed the codex/sqlite-plugin-install-index branch from 8f55198 to 66bcc17 Compare June 1, 2026 00:24

openclaw-barnacle Bot added the extensions: ollama label Jun 1, 2026

steipete added 13 commits June 1, 2026 01:41

refactor: persist plugin install index in sqlite

a7e7bca

fix: merge legacy plugin index records into sqlite

c6ed8d1

test: update plugin index sqlite fixtures

fade357

fix: migrate custom plugin install indexes

3887af9

test: update plugin index sentinel

f0c92e0

fix: exclude migrated plugin index archives

11a94d4

fix: read post-upgrade plugin index from sqlite

ed955f6

fix: migrate legacy plugin index before agent runs

771d03a

fix: respect disabled persisted plugin registry reads

cc4375f

test: type plugin install record fixtures

1cf6490

fix: simplify plugin index record reader type

8cada98

test: fix sqlite plugin index CI fallout

399eaec

test: mock provider normalization in agent command tests

a672738

# Conflicts: # src/commands/agent-command.test-mocks.ts

steipete force-pushed the codex/sqlite-plugin-install-index branch from 8e64f19 to a672738 Compare June 1, 2026 00:43

openclaw-barnacle Bot removed the extensions: ollama label Jun 1, 2026

build: remove unused ui three dependency

4d68180

openclaw-barnacle Bot added the app: web-ui App: web-ui label Jun 1, 2026

github-actions Bot added the dependencies-changed PR changes dependency-related files label Jun 1, 2026

steipete merged commit 5443baa into main Jun 1, 2026
34 of 37 checks passed

steipete deleted the codex/sqlite-plugin-install-index branch June 1, 2026 00:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Persist plugin install index in SQLite#88794

Persist plugin install index in SQLite#88794
steipete merged 14 commits into
mainfrom
codex/sqlite-plugin-install-index

steipete commented May 31, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented May 31, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 31, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026 •

edited

Loading

Uh oh!

steipete commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

steipete commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Uh oh!

clawsweeper Bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 1, 2026

Dependency Guard

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency graph change authorized

Uh oh!

steipete commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

steipete commented May 31, 2026 •

edited

Loading

clawsweeper Bot commented May 31, 2026 •

edited

Loading

github-actions Bot commented Jun 1, 2026 •

edited

Loading