Skip to content

Add marketplace feed refresh command#96155

Merged
giodl73-repo merged 1 commit into
mainfrom
feeds-hosted-catalog-marketplace-refresh
Jun 28, 2026
Merged

Add marketplace feed refresh command#96155
giodl73-repo merged 1 commit into
mainfrom
feeds-hosted-catalog-marketplace-refresh

Conversation

@giodl73-repo

@giodl73-repo giodl73-repo commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Hosted Marketplace Feed Stack

This is PR7 in the hosted marketplace feed stack. The stack should land in order, with maintainer review at each step before the later PRs wire hosted marketplace data into broader user flows.

  1. Refactor external plugin catalog toward feeds #95846: refactor the bundled external plugin catalog toward the feed-shaped data model.
  2. Add hosted external catalog feed loader #95868: add the lazy hosted marketplace feed loader for the configured public feed URL, with guarded HTTPS fetch, checksum, schema, size, timeout, and bundled fallback behavior.
  3. Add hosted catalog snapshot fallback #95877: add accepted hosted snapshot fallback so a failed or unchanged hosted fetch can reuse the last valid feed payload.
  4. Persist hosted catalog snapshots in state #95964: persist hosted marketplace feed snapshots in OpenClaw state.
  5. Add hosted catalog source profile validation #95969: add marketplace source profile validation for hosted feed install candidates.
  6. Add hosted catalog config profiles #95981: add configurable marketplaces.feeds and marketplaces.sources profiles.
  7. Add marketplace feed refresh command #96155: add openclaw plugins marketplace refresh as the explicit manual snapshot refresh path.
  8. Add marketplace feed entries command #96158: add openclaw plugins marketplace entries as the explicit manual read path over hosted, snapshot, or bundled fallback entries.
  9. Add marketplace feed telemetry #96194: add marketplace feed telemetry.

The broader follow-up after this stack is to decide whether and how marketplace entries should feed install/search/startup behavior, add automatic conditional refresh using ETag / Last-Modified, or move to ClawHub producer-side validation for /v1/feeds/plugins and /v1/feeds/skills.

Summary

Adds openclaw plugins marketplace refresh, a manual activation slice for the hosted marketplace feed stack. The command reads config.marketplaces, calls the configured hosted external plugin catalog loader, writes the accepted snapshot, and reports whether the accepted catalog came from the hosted feed, the last-known-good snapshot, or bundled fallback.

This gives maintainers a safe way to exercise the live https://clawhub.ai/v1/feeds/plugins path and any locally configured marketplaces.feeds profile without changing startup, onboarding, doctor, install, search, or provider-repair behavior yet.

What Changed

  • Adds plugins marketplace refresh with:
    • --feed-profile <name> for a configured marketplace feed profile.
    • --feed-url <url> for the existing guarded explicit hosted URL path.
    • --expected-sha256 <hash> for checksum-pinned refresh checks.
    • --json for automation-friendly source/feed/metadata output.
  • Keeps the command lazy-loaded with the existing plugin runtime boundary.
  • Reports hosted, hosted-snapshot, and bundled-fallback results.
  • Fails checksum-pinned refreshes when the accepted result is not a fresh hosted payload, so automation cannot silently accept bundled fallback after a checksum mismatch.
  • Normalizes bare and prefixed SHA-256 pins to the loader checksum format.
  • Rejects explicit hosted feed URLs with credentials, query strings, or fragments before fetch/snapshot persistence.
  • Keeps direct default ClawHub URL refresh behavior aligned with the default profile, while requiring manifest sourceRef for non-default direct feed URLs.
  • Updates the plugin CLI reference docs for plugins marketplace refresh.
  • Adds focused tests for CLI routing, refresh output, fallback reporting, redacted feed URL output, checksum normalization, URL rejection, source-ref enforcement, and pinned-refresh failure.

Validation

  • pnpm exec oxfmt --check src/cli/plugins-cli.runtime.ts src/cli/plugins-cli.marketplace-refresh.test.ts src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.ts docs/cli/plugins.md
  • pnpm exec oxlint --deny-warnings src/cli/plugins-cli.runtime.ts src/cli/plugins-cli.marketplace-refresh.test.ts src/plugins/official-external-plugin-catalog.ts src/plugins/official-external-plugin-catalog.test.ts
  • git diff --check
  • $env:CI='1'; node scripts/test-projects.mjs src/cli/plugins-cli.lazy.test.ts src/cli/plugins-cli.marketplace-refresh.test.ts src/plugins/official-external-plugin-catalog.test.ts (8 CLI refresh/lazy tests and 51 catalog tests passed)
  • pnpm docs:check-mdx
  • pnpm tsgo:core
  • codex review --base origin/main (no actionable correctness issues)

Note: pnpm format:docs:check still fails on this Windows host before checking this file because the repo-wide docs formatter builds an oxfmt command line that exceeds the Windows command-line length limit. The targeted oxfmt --check docs/cli/plugins.md and pnpm docs:check-mdx passed.

Real behavior proof

Behavior or issue addressed:
openclaw plugins marketplace refresh is the first manual activation path for hosted marketplace feeds. It accepts a fresh hosted payload and writes the accepted snapshot without enabling install/search/startup consumption.

Real environment tested:
Windows checkout C:\src\openclaw-prflow\.worktrees\pr-96155 at pushed PR7 head 1614070e92 (runtime proof path unchanged from 034ab82b95; current head adds docs-only refresh-behavior clarification), source CLI runtime harness via node --import tsx, isolated state dir C:\tmp\openclaw-feeds-proof\pr96155-refresh-034ab82, live HTTPS request to https://clawhub.ai/v1/feeds/plugins.

Exact steps or command run after this patch:

  1. Set runtime config snapshot with marketplaces.feeds.clawhub-public.url = https://clawhub.ai/v1/feeds/plugins.
  2. Set source profiles public-clawhub.type = clawhub and public-npm.type = npm.
  3. Ran runPluginMarketplaceRefreshCommand({ feedProfile: "clawhub-public", json: true }).
  4. Reused the same OPENCLAW_STATE_DIR and called loadConfiguredHostedOfficialExternalPluginCatalogEntries(config, { feedProfile: "clawhub-public", offline: true }) to verify the refresh wrote a durable snapshot.

Evidence after fix:

{
  "source": "hosted",
  "entries": 59,
  "metadata": {
    "url": "https://clawhub.ai/v1/feeds/plugins",
    "status": 200,
    "etag": "\"sha256:76a49aad745fe828c7f88b5a870ad67b5953c5235b4c928b1d6189584b257b63-gzip\"",
    "lastModified": "Sun, 28 Jun 2026 12:47:16 GMT",
    "checksum": "sha256:76a49aad745fe828c7f88b5a870ad67b5953c5235b4c928b1d6189584b257b63"
  },
  "feed": {
    "id": "clawhub-official",
    "generatedAt": "2026-06-28T12:47:09.109Z",
    "sequence": 19
  }
}

Offline snapshot verification from the same state dir:

{
  "proofHead": "034ab82b95",
  "stateDir": "C:\\tmp\\openclaw-feeds-proof\\pr96155-refresh-034ab82",
  "offlineSource": "hosted-snapshot",
  "offlineEntries": 59,
  "offlineFeed": {
    "id": "clawhub-official",
    "sequence": 19
  },
  "offlineChecksum": "sha256:76a49aad745fe828c7f88b5a870ad67b5953c5235b4c928b1d6189584b257b63"
}

Observed result after fix:
The refresh command accepted the live hosted ClawHub payload, reported source: hosted, returned 59 entries, and persisted a snapshot that the loader could read back offline from the same state directory.

What was not tested:
This proof does not install plugins from the refreshed feed. Installation, search, startup consumption, feed auth, and signed-feed enforcement are later slices.

@clawsweeper

clawsweeper Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 28, 2026, 1:07 PM ET / 17:07 UTC.

Summary
The branch adds openclaw plugins marketplace refresh with feed-profile, direct feed URL, checksum pinning, JSON/text output, hosted/snapshot/fallback reporting, docs, and focused tests.

PR surface: Source +236, Tests +478, Docs +19. Total +733 across 7 files.

Reproducibility: not applicable. This PR adds a new manual CLI capability rather than fixing an existing current-main bug. Current main and v2026.6.10 lack the refresh command, while the PR body includes live-output proof for the new path.

Review metrics: 1 noteworthy metric.

  • Public CLI Surface: 1 command added, 4 options added. The command and flags become compatibility-sensitive user and automation surface once merged.

Stored data model
Persistent data-model change detected: database schema: src/cli/plugins-cli.marketplace-refresh.test.ts, database schema: src/plugins/official-external-plugin-catalog.test.ts, serialized state: src/plugins/official-external-plugin-catalog.test.ts, vector/embedding metadata: src/cli/plugins-cli.runtime.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #96155
Summary: This PR is the canonical manual refresh-command slice in the hosted marketplace feed stack; related items are merged prerequisites or later read/telemetry slices, not replacements.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] Merging creates a public plugins marketplace refresh command and four flags that users or automation may depend on once released.
  • [P1] Successful hosted refreshes intentionally require snapshot persistence, so state database write failures become operator-visible command failures for this manual path.
  • [P2] Unpinned refreshes intentionally treat hosted-snapshot or bundled-fallback results as command success, which maintainers should explicitly accept as the manual refresh UX.

Maintainer options:

  1. Accept Refresh CLI Contract (recommended)
    Maintainers can land this slice after explicitly accepting the command name, four flags, JSON output shape, fallback modes, URL restrictions, snapshot-write failure behavior, and pinned-checksum semantics as public manual refresh API.
  2. Tighten Docs First
    Maintainers can require the docs to spell out fallback success, pinned-refresh failure, URL redaction, and snapshot-write failure behavior before the command ships.
  3. Pause For Marketplace UX
    If manual refresh is not the desired product surface, pause this slice and fold refresh into the later marketplace read, install, startup, or producer-side feed-validation design.

Next step before merge

  • [P2] Maintainer review is required because the PR has the protected maintainer label and adds a public CLI compatibility surface; no narrow automated repair defect was found.

Security
Cleared: Cleared: the diff adds no dependencies, workflows, lockfiles, or package execution paths, keeps fetches on the existing guarded loader path, and rejects or redacts sensitive feed URL components.

Review details

Best possible solution:

Land this slice only after maintainers accept the command name, flags, JSON shape, fallback modes, URL restrictions, and snapshot-write failure behavior as the supported manual hosted-feed refresh API.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this PR adds a new manual CLI capability rather than fixing an existing current-main bug. Current main and v2026.6.10 lack the refresh command, while the PR body includes live-output proof for the new path.

Is this the best way to solve the issue?

Yes as an implementation slice: it reuses the existing hosted loader, source-profile checks, SSRF guard, and snapshot store instead of adding a parallel fetch path. The remaining question is maintainer acceptance of the public CLI contract and fallback UX.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 891096926e27.

Label changes

Label justifications:

  • P2: This is normal-priority hosted marketplace CLI feature work with limited immediate blast radius but meaningful public API review surface.
  • merge-risk: 🚨 compatibility: The PR adds a public CLI command and flags over hosted marketplace feed snapshot state that users or automation may later depend on.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Sufficient: the PR body includes after-fix live output from a Windows source runtime harness that fetched the live ClawHub feed, wrote the accepted snapshot, and read it back offline from the same state directory.
  • proof: sufficient: Contributor real behavior proof is sufficient. Sufficient: the PR body includes after-fix live output from a Windows source runtime harness that fetched the live ClawHub feed, wrote the accepted snapshot, and read it back offline from the same state directory.
Evidence reviewed

PR surface:

Source +236, Tests +478, Docs +19. Total +733 across 7 files.

View PR surface stats
Area Files Added Removed Net
Source 3 237 1 +236
Tests 3 487 9 +478
Docs 1 19 0 +19
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 7 743 10 +733

What I checked:

  • Root policy read: Read the full root AGENTS.md; its public CLI, plugin boundary, compatibility, fallback, and review-depth guidance applies to this PR. (AGENTS.md:1, 891096926e27)
  • Scoped plugin policy read: Read src/plugins/AGENTS.md; the hosted external catalog path is under the plugin discovery/catalog boundary and should preserve lazy, contract-aligned behavior. (src/plugins/AGENTS.md:1, 891096926e27)
  • Scoped docs policy read: Read docs/AGENTS.md; the CLI doc addition follows the docs-owned plugin terminology and root-relative docs-link expectations. Public docs: docs/AGENTS.md. (docs/AGENTS.md:1, 891096926e27)
  • Current main behavior: Current main registers plugins marketplace list but no plugins marketplace refresh, so the PR is not already implemented on main. (src/cli/plugins-cli.ts:275, 891096926e27)
  • Latest release behavior: The latest release tag v2026.6.10 also only has plugins marketplace list, so the refresh command has not shipped in the latest release. (src/cli/plugins-cli.ts:246, aa69b12d0086)
  • PR command registration: PR head registers marketplace refresh and its four public options before lazy-loading the plugin runtime. (src/cli/plugins-cli.ts:286, 1614070e92cd)

Likely related people:

  • giodl73-repo: Authored the merged hosted feed loader, snapshot fallback, persistent snapshot store, source-profile validation, and marketplace config-profile commits that this refresh command builds on. (role: recent area contributor and feature owner; confidence: high; commits: c29e1fe764f3, 12685ee6b752, d1b917120a47; files: src/plugins/official-external-plugin-catalog.ts, src/plugins/official-external-plugin-catalog-snapshot-store.ts, src/plugins/official-external-plugin-catalog.test.ts)
  • llagy009: The shallow current-main blame for the existing plugins marketplace list command block points to a broad recent commit, making this a low-confidence CLI-adjacent routing signal only. (role: adjacent CLI contributor; confidence: low; commits: b5c662f4f54c; files: src/cli/plugins-cli.ts)
  • Patrick-Erichsen: Provided marketplace naming/API-shape feedback on the predecessor marketplace config PR that this refresh command consumes through config.marketplaces. (role: API naming reviewer; confidence: medium; files: src/config/types.marketplaces.ts, src/plugins/official-external-plugin-catalog.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels Jun 24, 2026
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-config-profiles branch from fdd7f78 to 05a3e13 Compare June 24, 2026 02:48
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from 76baf5a to 9dcb7b7 Compare June 24, 2026 02:48
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-config-profiles branch from 05a3e13 to d1f3927 Compare June 24, 2026 03:51
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from 9dcb7b7 to 0e4486b Compare June 24, 2026 03:51
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

Addressed the marketplace feed URL output concern in this slice: explicit feed URLs with userinfo are rejected before fetch, and marketplace refresh JSON/text output now redacts query strings, fragments, username, and password from emitted feed URLs and fallback errors. Added focused regression coverage. @clawsweeper re-review

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jun 24, 2026
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-config-profiles branch from d1f3927 to c7565c0 Compare June 24, 2026 13:45
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from 0e4486b to 15b033a Compare June 24, 2026 13:45
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-config-profiles branch from c7565c0 to da81650 Compare June 24, 2026 14:06
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from 15b033a to 82c4042 Compare June 24, 2026 14:07
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from b93aa0e to 219aa1d Compare June 28, 2026 15:29
@openclaw-barnacle openclaw-barnacle Bot added size: L and removed scripts Repository scripts size: XL labels Jun 28, 2026
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@giodl73-repo
giodl73-repo marked this pull request as ready for review June 28, 2026 15:35
@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from 219aa1d to 034ab82 Compare June 28, 2026 16:15
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jun 28, 2026
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@giodl73-repo
giodl73-repo force-pushed the feeds-hosted-catalog-marketplace-refresh branch from 034ab82 to 1614070 Compare June 28, 2026 16:57
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@giodl73-repo
giodl73-repo merged commit 102a654 into main Jun 28, 2026
101 checks passed
@giodl73-repo
giodl73-repo deleted the feeds-hosted-catalog-marketplace-refresh branch June 28, 2026 17:29
@giodl73-repo

Copy link
Copy Markdown
Contributor Author

Merged via squash.

Thanks @giodl73-repo!

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 29, 2026
Merged via squash.

Prepared head SHA: 1614070
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
Merged via squash.

Prepared head SHA: 1614070
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
Merged via squash.

Prepared head SHA: 1614070
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
Merged via squash.

Prepared head SHA: 1614070
Co-authored-by: giodl73-repo <[email protected]>
Co-authored-by: giodl73-repo <[email protected]>
Reviewed-by: @giodl73-repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli CLI command changes docs Improvements or additions to documentation maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant