Add hosted external catalog feed loader#95868
Conversation
c8012b2 to
2acadea
Compare
|
Maybe instead of https://register.openclaw.ai/official-external-plugin-catalog.json we go with something more generic that can expand to include skills, eg https://register.openclaw.ai/official-marketplace.json ? |
2acadea to
f184d6d
Compare
|
Updated this slice to use the generic marketplace endpoint Patrick suggested: https://register.openclaw.ai/official-marketplace.json. I kept the bundled fallback file name unchanged so this PR stays focused on the public hosted endpoint contract. |
|
Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 4:15 PM ET / 20:15 UTC. Summary PR surface: Source +323, Tests +313. Total +636 across 2 files. Reproducibility: not applicable. This PR adds a new hosted marketplace feed transport path rather than fixing a reproduced bug. Source inspection and the PR body's live ClawHub feed output confirm the intended after-change behavior path. Review metrics: 1 noteworthy metric.
Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Keep this as the isolated hosted transport slice, merge only after maintainers accept the default ClawHub feed URL, hostname allowlist, schema acceptance, checksum metadata, and bundled-fallback contract, and leave install/search/startup wiring to the follow-up PRs. Do we have a high-confidence way to reproduce the issue? Not applicable; this PR adds a new hosted marketplace feed transport path rather than fixing a reproduced bug. Source inspection and the PR body's live ClawHub feed output confirm the intended after-change behavior path. Is this the best way to solve the issue? Yes, as an isolated transport-layer slice with a maintainer decision still required. The implementation stays in the plugin catalog control-plane module, keeps the fetch guard lazy, and avoids startup/install wiring while the remaining question is acceptance of the remote feed boundary. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against c52adf7505a7. Label changesLabel justifications:
Evidence reviewedPR surface: Source +323, Tests +313. Total +636 across 2 files. View PR surface stats
Security concerns:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Merged via squash.
Thanks @giodl73-repo! |
Merged via squash. Prepared head SHA: 76da932 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Merged via squash. Prepared head SHA: 76da932 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Merged via squash. Prepared head SHA: 76da932 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Merged via squash. Prepared head SHA: 76da932 Co-authored-by: giodl73-repo <[email protected]> Co-authored-by: giodl73-repo <[email protected]> Reviewed-by: @giodl73-repo
Hosted Marketplace Feed Stack
This is PR2 in the hosted marketplace feed stack. The stack should land in order, with maintainer review at each step before the later PRs wire hosted marketplace data into broader user flows.
marketplaces.feedsandmarketplaces.sourcesprofiles.openclaw plugins marketplace refreshas the explicit manual snapshot refresh path.openclaw plugins marketplace entriesas the explicit manual read path over hosted, snapshot, or bundled fallback entries.The broader follow-up after this stack is to decide whether and how marketplace entries should feed install/search/startup behavior, add automatic conditional refresh using
ETag/Last-Modified, or move to ClawHub producer-side validation for/v1/feeds/pluginsand/v1/feeds/skills.Journey Context
This is step 2 of the hosted OpenClaw feed path from RFC 19. #95846 made the bundled catalog feed-shaped. This PR adds the guarded hosted fetch path for that same marketplace feed, while leaving the existing synchronous bundled-catalog startup path unchanged.
What This PR Does
If-None-MatchandIf-Modified-Sinceheaders so callers can perform freshness checks.sha256metadata for the hosted response body.What This PR Intentionally Does Not Do
This PR does not wire the hosted loader into startup, add persistent snapshots, add source-profile selection, interpret
install.candidates[].integrity, add auth, add signed envelopes, add regional feeds, or change plugin install/search behavior. #95877 adds the loader-level last-known-good snapshot contract on top of this fetch path, and #95969 owns source-profile validation plus ClawHub checksum semantics.How To Review
Review this as the hosted transport and validation layer. The important contract is: a hosted marketplace feed can be tried safely, but any unsafe or unusable hosted result falls back to the bundled catalog instead of changing current behavior.
Maintainer acceptance requested: this slice establishes the default ClawHub feed URL, hostname allowlist, accepted feed schema versions, checksum metadata, and bundled-fallback transport contract. It deliberately keeps install interpretation, snapshot persistence, config, and search/startup wiring in follow-up PRs.
Validation
pnpm exec oxfmt --check src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.tspnpm exec oxlint --deny-warnings src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.tsgit diff --checkCI=1 node scripts/test-projects.mjs src/plugins/official-external-plugin-catalog.test.ts(26 tests)pnpm tsgo:corecodex review --base origin/mainafter the metadata-only dedupe fix: no actionable correctness issuescodex review --base origin/mainafter pushing head76da9328af8498b99f133f8f0fe3f10f42cbee3d: no actionable correctness issuesscripts/pr review-validate-artifacts 95868,OPENCLAW_TESTBOX=1 scripts/pr prepare-gates 95868,OPENCLAW_TESTBOX=1 scripts/pr prepare-push 95868Real Behavior Proof
Behavior or issue addressed:
The hosted catalog fetch path uses the live ClawHub plugin feed by default, accepts the live feed wrapper, preserves raw marketplace package metadata without synthesizing OpenClaw install metadata, and still falls back safely on invalid hosted responses.
Real environment tested:
Windows checkout
C:\src\openclaw-prflow\.worktrees\pr-95868at pushed head76da9328af8498b99f133f8f0fe3f10f42cbee3d, source harness vianode --import tsx, live HTTPS request tohttps://clawhub.ai/v1/feeds/plugins.Exact steps or command run after this patch:
node --import tsx -e "... loadHostedOfficialExternalPluginCatalogEntries({ snapshotStore: null }) ..."pnpm exec oxlint --deny-warnings src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.tspnpm exec oxfmt --check src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.tsgit diff --checkCI=1 node scripts/test-projects.mjs src/plugins/official-external-plugin-catalog.test.tscodex review --base origin/mainEvidence after fix:
{ "head": "76da9328af8498b99f133f8f0fe3f10f42cbee3d", "defaultFeedUrl": "https://clawhub.ai/v1/feeds/plugins", "resultSource": "hosted", "metadata": { "url": "https://clawhub.ai/v1/feeds/plugins", "status": 200, "checksum": "sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d", "etag": "\"sha256:cf3d85c1d6ede1907ca265e9b214ac6ff1c715c90c30ec65e8017addb590e32d-gzip\"", "lastModified": "Sat, 27 Jun 2026 18:43:36 GMT" }, "entryCount": 59, "feed": { "id": "clawhub-official", "sequence": 16, "schemaVersion": 1 }, "firstEntries": [ { "id": "@expediagroup/expedia-openclaw", "label": "plugin", "npmSpec": "@expediagroup/expedia-openclaw", "state": "available", "type": "plugin" }, { "id": "@openclaw/acpx", "label": "plugin", "npmSpec": "@openclaw/acpx", "state": "available", "type": "plugin" }, { "id": "@openclaw/amazon-bedrock-mantle-provider", "label": "plugin", "npmSpec": "@openclaw/amazon-bedrock-mantle-provider", "state": "available", "type": "plugin" } ] }Observed result after fix:
The default hosted fetch accepted the live ClawHub feed as
hosted, returned 59 hosted entries, preserved response checksum/validators, and left marketplace candidates as raw feed metadata so #95969 owns source-profile install interpretation.What was not tested:
Snapshot reuse and candidate-to-install interpretation are not part of this PR. Snapshot reuse is covered by #95877; source-profile install interpretation is covered by #95969. Maintainer acceptance is still needed for the default ClawHub feed URL, hostname allowlist, schema acceptance, checksum metadata, and bundled-fallback transport contract.