Persist hosted catalog snapshots in state#95964
Conversation
b1cb838 to
b486def
Compare
c371b39 to
f261e76
Compare
|
Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 9:20 PM ET / 01:20 UTC. Summary PR surface: Source +188, Tests +111, Generated +26. Total +325 across 6 files. Reproducibility: not applicable. this is a storage feature slice, not a reported bug. The PR body and focused tests give a high-confidence proof path for hosted fetch to SQLite snapshot to HTTP 304 reuse. Review metrics: 2 noteworthy metrics.
Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land only after maintainers explicitly accept the shared-state persistence default and confirm persisted snapshots remain inert cache data until the source-profile and trust slices land. Do we have a high-confidence way to reproduce the issue? Not applicable; this is a storage feature slice, not a reported bug. The PR body and focused tests give a high-confidence proof path for hosted fetch to SQLite snapshot to HTTP 304 reuse. Is this the best way to solve the issue? Yes, if maintainers accept the boundary: shared SQLite state matches repository storage policy and AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 3630d502eb9b. Label changesLabel justifications:
Evidence reviewedPR surface: Source +188, Tests +111, Generated +26. Total +325 across 6 files. View PR surface stats
Security concerns:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
f261e76 to
fb61b1d
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
68a85a8 to
be8678a
Compare
fb61b1d to
13b94e7
Compare
49c116f to
372ec63
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Merged via squash.
Thanks @giodl73-repo! |
Merged via squash. Prepared head SHA: 372ec63 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Merged via squash. Prepared head SHA: 372ec63 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Merged via squash. Prepared head SHA: 372ec63 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Merged via squash. Prepared head SHA: 372ec63 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Hosted Marketplace Feed Stack
This is PR4 in the hosted marketplace feed stack. The stack should land in order, with maintainer review at each step before the later PRs wire hosted marketplace data into broader user flows.
marketplaces.feedsandmarketplaces.sourcesprofiles.openclaw plugins marketplace refreshas the explicit manual snapshot refresh path.openclaw plugins marketplace entriesas the explicit manual read path over hosted, snapshot, or bundled fallback entries.The broader follow-up after this stack is to decide whether and how marketplace entries should feed install/search/startup behavior, add automatic conditional refresh using
ETag/Last-Modified, or move to ClawHub producer-side validation for/v1/feeds/pluginsand/v1/feeds/skills.Stack Context
This is step 4 in the hosted feed rollout. It has been restacked directly on current
mainafter #95846, #95868, and #95877 merged, so the diff is limited to the persistent snapshot storage layer and its tests.RFC Alignment
This PR is the persistent snapshot plumbing for the hosted catalog loader. It implements the SQLite storage part of RFC #19 rollout item 6: store the latest accepted hosted feed snapshot in OpenClaw state so the client can keep using a last-known-good catalog during HTTP 304 responses or transient hosted-feed failures.
It intentionally does not claim the full lifecycle refresh implementation. Named feed/source profiles, source-ref validation, signed envelopes, scheduled refresh,
maxStale, and feed freshness reporting are still follow-up PRs.Maintainer Boundary Decision
Maintainer review accepts this PR as the durable cache layer only: persisted hosted feed bodies are inert snapshot material, not trusted install, search, startup, or source authority. Later stack PRs must explicitly validate source profiles and feed trust rules before consuming persisted entries for marketplace behavior.
The default shared-state SQLite store is intentional for hosted loader resilience after HTTP 304 or transient feed failures.
snapshotStore: nullremains the explicit opt-out for callers and tests that must avoid persistence.Summary
snapshotStore: nullfor callers and tests that need persistence disabled.Deliberately Not Included
sourceRefvalidation against configured source profiles.maxStale, or stale-catalog status reporting.Those remain follow-up steps after the core fallback storage path is reviewable.
Validation
pnpm exec oxlint --deny-warnings src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.ts src/plugins/official-external-plugin-catalog-snapshot-store.ts src/state/openclaw-state-schema.generated.tspnpm exec oxfmt --check src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.ts src/plugins/official-external-plugin-catalog-snapshot-store.ts src/state/openclaw-state-schema.generated.ts src/state/openclaw-state-db.generated.d.tspnpm db:kysely:checkpnpm lint:kyselypnpm check:database-first-legacy-storespnpm check:architecturepnpm tsgo:coregit diff --checkCI=1 node scripts/test-projects.mjs src/plugins/official-external-plugin-catalog.test.ts- 35 tests passedcodex review --base origin/main- no actionable correctness issues foundReal behavior proof
Behavior or issue addressed:
This PR persists accepted hosted marketplace feed snapshots in OpenClaw SQLite state so the hosted catalog loader can reuse a last-known-good payload after live HTTP 304 responses or transient hosted-feed failures.
Real environment tested:
Windows host,
C:\src\openclaw-prflow\.worktrees\pr-95964, source harness on commit372ec63c99, Node withtsx, isolated state directoryC:\\tmp\\openclaw-feeds-proof\\pr95964-live-snapshot-kysely, and the live ClawHub feed endpointhttps://clawhub.ai/v1/feeds/plugins.Exact steps or command run after this patch:
loadHostedOfficialExternalPluginCatalogEntries({ stateDir, timeoutMs: 30000, chunkTimeoutMs: 30000 })against the live ClawHub feed.ifNoneMatchset to the live ETag returned by step 2.createSqliteHostedOfficialExternalPluginCatalogSnapshotStore({ stateDir })and checkedstate/openclaw.sqliteon disk.Evidence after fix:
The live harness printed:
{ "repoHead": "372ec63c99", "stateDir": "C:/tmp/openclaw-feeds-proof/pr95964-live-snapshot-kysely", "sqlitePath": "C:\\\\tmp\\\\openclaw-feeds-proof\\\\pr95964-live-snapshot-kysely\\\\state\\\\openclaw.sqlite", "sqliteBytes": 4096, "firstLoad": { "source": "hosted", "feedId": "clawhub-official", "schemaVersion": 1, "sequence": 16, "entries": 59, "etag": "\"sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d-gzip\"", "lastModified": "Sat, 27 Jun 2026 18:43:36 GMT", "checksum": "sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d" }, "persistedSnapshot": { "feedUrl": "https://clawhub.ai/v1/feeds/plugins", "status": 200, "checksum": "sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d", "savedAt": "2026-06-28T00:08:40.573Z", "bodyBytes": 21942 }, "secondConditionalLoad": { "source": "hosted-snapshot", "entries": 59, "feedId": "clawhub-official", "sequence": 16, "error": "hosted catalog feed returned HTTP 304", "status": 200, "etag": "\"sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d-gzip\"" } }Observed result after fix:
The first live load fetched the hosted feed and created the SQLite state database. The second conditional load received HTTP 304 from ClawHub and returned
source: "hosted-snapshot"with the same 59 entries from the persisted SQLite snapshot.What was not tested:
This proof did not exercise a packaged OpenClaw binary or CLI command path. The source harness covered the hosted loader's default persistent store wiring, and the focused tests cover snapshot validation, opt-out behavior, schema shape, and transient failure fallback.