Skip to content

Persist hosted catalog snapshots in state#95964

Merged
giodl73-repo merged 2 commits into
mainfrom
feeds-hosted-catalog-persistent-snapshot
Jun 28, 2026
Merged

Persist hosted catalog snapshots in state#95964
giodl73-repo merged 2 commits into
mainfrom
feeds-hosted-catalog-persistent-snapshot

Conversation

@giodl73-repo

@giodl73-repo giodl73-repo commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Hosted Marketplace Feed Stack

This is PR4 in the hosted marketplace feed stack. The stack should land in order, with maintainer review at each step before the later PRs wire hosted marketplace data into broader user flows.

  1. Refactor external plugin catalog toward feeds #95846: refactor the bundled external plugin catalog toward the feed-shaped data model. Merged.
  2. Add hosted external catalog feed loader #95868: add the lazy hosted marketplace feed loader for the configured public feed URL, with guarded HTTPS fetch, checksum, schema, size, timeout, and bundled fallback behavior. Merged.
  3. Add hosted catalog snapshot fallback #95877: add accepted hosted snapshot fallback so a failed or unchanged hosted fetch can reuse the last valid feed payload. Merged.
  4. Persist hosted catalog snapshots in state #95964: persist hosted marketplace feed snapshots in OpenClaw state. This PR.
  5. Add hosted catalog source profile validation #95969: add marketplace source profile validation for hosted feed install candidates.
  6. Add hosted catalog config profiles #95981: add configurable marketplaces.feeds and marketplaces.sources profiles.
  7. Add marketplace feed refresh command #96155: add openclaw plugins marketplace refresh as the explicit manual snapshot refresh path.
  8. Add marketplace feed entries command #96158: add openclaw plugins marketplace entries as the explicit manual read path over hosted, snapshot, or bundled fallback entries.
  9. Add marketplace feed telemetry #96194: add marketplace feed telemetry.

The broader follow-up after this stack is to decide whether and how marketplace entries should feed install/search/startup behavior, add automatic conditional refresh using ETag / Last-Modified, or move to ClawHub producer-side validation for /v1/feeds/plugins and /v1/feeds/skills.

Stack Context

This is step 4 in the hosted feed rollout. It has been restacked directly on current main after #95846, #95868, and #95877 merged, so the diff is limited to the persistent snapshot storage layer and its tests.

RFC Alignment

This PR is the persistent snapshot plumbing for the hosted catalog loader. It implements the SQLite storage part of RFC #19 rollout item 6: store the latest accepted hosted feed snapshot in OpenClaw state so the client can keep using a last-known-good catalog during HTTP 304 responses or transient hosted-feed failures.

It intentionally does not claim the full lifecycle refresh implementation. Named feed/source profiles, source-ref validation, signed envelopes, scheduled refresh, maxStale, and feed freshness reporting are still follow-up PRs.

Maintainer Boundary Decision

Maintainer review accepts this PR as the durable cache layer only: persisted hosted feed bodies are inert snapshot material, not trusted install, search, startup, or source authority. Later stack PRs must explicitly validate source profiles and feed trust rules before consuming persisted entries for marketplace behavior.

The default shared-state SQLite store is intentional for hosted loader resilience after HTTP 304 or transient feed failures. snapshotStore: null remains the explicit opt-out for callers and tests that must avoid persistence.

Summary

  • Adds a SQLite-backed snapshot store for hosted official external plugin catalog responses in OpenClaw state.
  • Makes the hosted feed loader use that state-backed store by default while preserving snapshotStore: null for callers and tests that need persistence disabled.
  • Persists the feed body plus URL, status, ETag, Last-Modified, locally computed checksum, save time, and update time.
  • Lets HTTP 304 and transient hosted feed failures use the last known good state snapshot before falling back to the bundled catalog.
  • Keeps snapshots keyed by the resolved feed URL, which leaves room for later named feed/source profile work without mixing feeds.

Deliberately Not Included

  • Named local feed/source profile configuration.
  • sourceRef validation against configured source profiles.
  • Authenticated feed URLs or secret-reference resolution.
  • Signed feed envelopes or trust-key configuration.
  • Background refresh scheduling, maxStale, or stale-catalog status reporting.
  • Search, install UX, skills, or telemetry.

Those remain follow-up steps after the core fallback storage path is reviewable.

Validation

  • pnpm exec oxlint --deny-warnings src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.ts src/plugins/official-external-plugin-catalog-snapshot-store.ts src/state/openclaw-state-schema.generated.ts
  • pnpm exec oxfmt --check src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.ts src/plugins/official-external-plugin-catalog-snapshot-store.ts src/state/openclaw-state-schema.generated.ts src/state/openclaw-state-db.generated.d.ts
  • pnpm db:kysely:check
  • pnpm lint:kysely
  • pnpm check:database-first-legacy-stores
  • pnpm check:architecture
  • pnpm tsgo:core
  • git diff --check
  • CI=1 node scripts/test-projects.mjs src/plugins/official-external-plugin-catalog.test.ts - 35 tests passed
  • codex review --base origin/main - no actionable correctness issues found

Real behavior proof

Behavior or issue addressed:
This PR persists accepted hosted marketplace feed snapshots in OpenClaw SQLite state so the hosted catalog loader can reuse a last-known-good payload after live HTTP 304 responses or transient hosted-feed failures.

Real environment tested:
Windows host, C:\src\openclaw-prflow\.worktrees\pr-95964, source harness on commit 372ec63c99, Node with tsx, isolated state directory C:\\tmp\\openclaw-feeds-proof\\pr95964-live-snapshot-kysely, and the live ClawHub feed endpoint https://clawhub.ai/v1/feeds/plugins.

Exact steps or command run after this patch:

  1. Removed the isolated state directory.
  2. Ran a source harness that called loadHostedOfficialExternalPluginCatalogEntries({ stateDir, timeoutMs: 30000, chunkTimeoutMs: 30000 }) against the live ClawHub feed.
  3. Re-ran the same loader with ifNoneMatch set to the live ETag returned by step 2.
  4. Read the persisted snapshot through createSqliteHostedOfficialExternalPluginCatalogSnapshotStore({ stateDir }) and checked state/openclaw.sqlite on disk.

Evidence after fix:
The live harness printed:

{
  "repoHead": "372ec63c99",
  "stateDir": "C:/tmp/openclaw-feeds-proof/pr95964-live-snapshot-kysely",
  "sqlitePath": "C:\\\\tmp\\\\openclaw-feeds-proof\\\\pr95964-live-snapshot-kysely\\\\state\\\\openclaw.sqlite",
  "sqliteBytes": 4096,
  "firstLoad": {
    "source": "hosted",
    "feedId": "clawhub-official",
    "schemaVersion": 1,
    "sequence": 16,
    "entries": 59,
    "etag": "\"sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d-gzip\"",
    "lastModified": "Sat, 27 Jun 2026 18:43:36 GMT",
    "checksum": "sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d"
  },
  "persistedSnapshot": {
    "feedUrl": "https://clawhub.ai/v1/feeds/plugins",
    "status": 200,
    "checksum": "sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d",
    "savedAt": "2026-06-28T00:08:40.573Z",
    "bodyBytes": 21942
  },
  "secondConditionalLoad": {
    "source": "hosted-snapshot",
    "entries": 59,
    "feedId": "clawhub-official",
    "sequence": 16,
    "error": "hosted catalog feed returned HTTP 304",
    "status": 200,
    "etag": "\"sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d-gzip\""
  }
}

Observed result after fix:
The first live load fetched the hosted feed and created the SQLite state database. The second conditional load received HTTP 304 from ClawHub and returned source: "hosted-snapshot" with the same 59 entries from the persisted SQLite snapshot.

What was not tested:
This proof did not exercise a packaged OpenClaw binary or CLI command path. The source harness covered the hosted loader's default persistent store wiring, and the focused tests cover snapshot validation, opt-out behavior, schema shape, and transient failure fallback.

@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-snapshot-cache branch from b1cb838 to b486def Compare June 23, 2026 16:11
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-persistent-snapshot branch from c371b39 to f261e76 Compare June 23, 2026 16:11
@clawsweeper

clawsweeper Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 9:20 PM ET / 01:20 UTC.

Summary
The PR adds a Kysely-backed SQLite snapshot store for hosted official external plugin catalog responses, wires the hosted loader to use it by default with snapshotStore: null as opt-out, and updates schema/types/tests.

PR surface: Source +188, Tests +111, Generated +26. Total +325 across 6 files.

Reproducibility: not applicable. this is a storage feature slice, not a reported bug. The PR body and focused tests give a high-confidence proof path for hosted fetch to SQLite snapshot to HTTP 304 reuse.

Review metrics: 2 noteworthy metrics.

  • Stored Data Model: 1 SQLite table added. The PR adds shared-state persistence, so maintainers should notice the upgrade and local-state impact before merge.
  • Loader Persistence Default: 1 default changed. Hosted loader callers that omit snapshotStore now use shared-state persistence unless they explicitly opt out.

Stored data model
Persistent data-model change detected: database schema: src/state/openclaw-state-schema.generated.ts, database schema: src/state/openclaw-state-schema.sql, serialized state: src/state/openclaw-state-db.generated.d.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #95964
Summary: This PR is the canonical persistent-storage slice in the hosted marketplace feed stack; related items are prerequisites or follow-up slices rather than replacements.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] Omitted hosted-loader snapshotStore calls now create or use shared SQLite persistence by default after an accepted hosted response, so no-local-write callers must explicitly pass snapshotStore: null.
  • [P1] The PR stores unsigned hosted marketplace feed bodies in shared OpenClaw state before the follow-up source-profile and trust-rule slices decide how those entries may be consumed.
  • [P1] The new shared-state table is additive and follows the existing schema-open path, but it is still an upgrade-sensitive stored data-model change that needs maintainer acceptance.

Maintainer options:

  1. Land After Boundary Approval (recommended)
    Accept the shared-state persistence default and unsigned snapshot cache boundary as intentional once required checks remain green.
  2. Make Persistence Explicit
    Require callers to opt into SQLite persistence if maintainers want omitted snapshotStore calls to preserve no-local-write behavior.
  3. Pause For Trust Slices
    Defer this PR until source-profile validation and feed trust rules land if maintainers do not want unsigned feed bodies persisted first.

Next step before merge

  • [P2] The remaining action is maintainer acceptance of the shared-state persistence and trust boundary, not a narrow automation repair.

Security
Needs attention: Needs maintainer attention because the patch persists unsigned hosted marketplace feed bodies in shared state before follow-up trust/source validation consumes them.

Review details

Best possible solution:

Land only after maintainers explicitly accept the shared-state persistence default and confirm persisted snapshots remain inert cache data until the source-profile and trust slices land.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this is a storage feature slice, not a reported bug. The PR body and focused tests give a high-confidence proof path for hosted fetch to SQLite snapshot to HTTP 304 reuse.

Is this the best way to solve the issue?

Yes, if maintainers accept the boundary: shared SQLite state matches repository storage policy and snapshotStore: null preserves an explicit no-persistence path. The safer alternative is to make persistence opt-in or pause until source-profile and trust validation land.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3630d502eb9b.

Label changes

Label justifications:

  • P2: This is a normal-priority hosted marketplace persistence slice with bounded scope but maintainer-visible compatibility and trust-boundary impact.
  • merge-risk: 🚨 compatibility: The PR adds a shared state schema table and changes omitted snapshotStore calls to persist accepted hosted snapshots by default.
  • merge-risk: 🚨 security-boundary: The PR stores unsigned hosted marketplace feed bodies that later stack slices may consume for marketplace behavior.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix live output showing a first ClawHub hosted load persisted a SQLite snapshot and a second conditional load returned source: hosted-snapshot after HTTP 304.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live output showing a first ClawHub hosted load persisted a SQLite snapshot and a second conditional load returned source: hosted-snapshot after HTTP 304.
Evidence reviewed

PR surface:

Source +188, Tests +111, Generated +26. Total +325 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 3 196 8 +188
Tests 1 112 1 +111
Docs 0 0 0 0
Config 0 0 0 0
Generated 2 26 0 +26
Other 0 0 0 0
Total 6 334 9 +325

Security concerns:

  • [medium] Confirm unsigned feed snapshot boundary — src/plugins/official-external-plugin-catalog-snapshot-store.ts:121
    The new store writes accepted hosted feed bodies to shared SQLite state, so maintainers should explicitly accept that these snapshots remain inert cache data until later source-profile and trust validation slices consume them.
    Confidence: 0.82

What I checked:

Likely related people:

  • giodl73-repo: They authored and merged the prerequisite hosted-feed shape, hosted loader, and snapshot fallback PRs now present on main, and this PR builds directly on those current code paths. (role: feature owner and recent area contributor; confidence: high; commits: 4010b81a77f4, c29e1fe764f3, 12685ee6b752; files: src/plugins/official-external-plugin-catalog.ts, src/plugins/official-external-plugin-catalog.test.ts, src/plugins/official-external-plugin-catalog-snapshot-store.ts)
  • Josh Avant: Current blame for the shared state DB schema lifecycle and schema file that this PR extends points to their recent state database work. (role: adjacent state database contributor; confidence: medium; commits: cee2aca40912; files: src/state/openclaw-state-db.ts, src/state/openclaw-state-schema.sql, src/state/openclaw-state-schema.generated.ts)
  • Patrick-Erichsen: Live PR review metadata shows an approval on an earlier head, making them relevant review context after the later Kysely rewrite. (role: reviewer; confidence: medium; commits: c371b39a93e0; files: src/plugins/official-external-plugin-catalog.ts, src/state/openclaw-state-schema.sql)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jun 23, 2026
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-persistent-snapshot branch from f261e76 to fb61b1d Compare June 24, 2026 14:04
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 24, 2026
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-snapshot-cache branch from 68a85a8 to be8678a Compare June 26, 2026 05:26
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-persistent-snapshot branch from fb61b1d to 13b94e7 Compare June 26, 2026 05:26
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 26, 2026
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation app: macos App: macos scripts Repository scripts size: L and removed size: M labels Jun 28, 2026
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-persistent-snapshot branch from 49c116f to 372ec63 Compare June 28, 2026 01:12
@openclaw-barnacle openclaw-barnacle Bot added size: M and removed docs Improvements or additions to documentation app: macos App: macos scripts Repository scripts size: L labels Jun 28, 2026
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@giodl73-repo
giodl73-repo merged commit d1b9171 into main Jun 28, 2026
163 of 169 checks passed
@giodl73-repo
giodl73-repo deleted the feeds-hosted-catalog-persistent-snapshot branch June 28, 2026 02:01
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

Merged via squash.

Thanks @giodl73-repo!

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 28, 2026
Merged via squash.

Prepared head SHA: 372ec63
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
Merged via squash.

Prepared head SHA: 372ec63
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
Merged via squash.

Prepared head SHA: 372ec63
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
Merged via squash.

Prepared head SHA: 372ec63
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants