fix(plugin-sdk): guard provider catalog live URL parsing against malformed responses by LZY3538 · Pull Request #109986 · openclaw/openclaw

LZY3538 · 2026-07-17T12:36:12Z

What Problem This Solves

Fixes an issue where fetchLiveProviderModelRows / fetchLiveProviderModelIds would crash with an uncaught TypeError when a provider model catalog API returns a genuinely malformed absolute URL in its next / links.next pagination field, or when the guarded fetch reports an unparseable redirect target (finalUrl).

Why This Change Was Made

resolveLiveModelCatalogNextPage and fetchLiveProviderModelRows each called new URL() on external API response data without try/catch. The fix introduces a tryParseUrl helper that returns undefined on parse failures, matching the defensive pattern already used by withoutMatrixStateAfterSyncParam in the Matrix transport (#109759).

When a provider-advertised next URL is malformed or cross-origin, the fix attempts cursor-based pagination as a fallback, then returns the controlled incomplete result when no fallback remains — preventing silent catalog truncation.

User Impact

Provider live model discovery survives misbehaving catalog APIs without crashing; a malformed pagination URL surfaces a controlled incomplete-pagination error instead of an uncaught TypeError or a silently truncated catalog.

Evidence

Before (upstream main — TypeError crash through real guarded fetch):

{"ok":false,"error":"TypeError: Invalid URL","label":"before-fix"}

After (this branch — controlled incomplete-pagination error through real guarded fetch):

{"ok":false,"error":"Error: proof model discovery did not include a supported next page before the catalog completed","label":"after-fix"}

The harness drives the real exported fetchLiveProviderModelIds through a loopback HTTP server using the production fetchWithSsrFGuard and ssrfPolicyFromHttpBaseUrlAllowedHostname. The malformed next URL contains a space in the hostname (http://exa mple.com/p2), which is a genuinely invalid absolute URL that triggers the native new URL() parse failure.

Focused tests (24 passed):

$ node scripts/run-vitest.mjs run src/plugin-sdk/provider-catalog-live-runtime.test.ts
 Test Files  1 passed (1)
      Tests  24 passed (24)

New tests cover: malformed absolute next URL (incomplete-pagination error), malformed nested links.next URL, cursor fallback recovery from malformed URL, unparseable finalUrl (cross-origin replay header fallback), and malformed next with has_more: true.

What was not tested: Live provider catalog APIs returning malformed pagination URLs; the loopback harness proves the defensive parsing guard on the real exported function through the production guarded-HTTP stack.

🤖 Generated with Claude Code

clawsweeper · 2026-07-17T12:44:59Z

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 9:36 AM ET / 13:36 UTC.

Summary
The PR adds non-throwing URL parsing for live provider-catalog pagination and redirect metadata, uses cursor fallback before reporting incomplete pagination, and adds malformed-response regressions.

PR surface: Source +32, Tests +150. Total +182 across 2 files.

Reproducibility: yes. The PR provides a high-confidence loopback reproduction through the exported catalog function and production guarded-fetch stack, showing the current-main TypeError and the branch's controlled incomplete result.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/plugin-sdk/provider-catalog-live-runtime.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] This public plugin-SDK helper changes unusable cross-origin or malformed next links from possible partial completion to a controlled discovery failure, so a nonconforming provider may fall back to its static catalog after upgrade instead of exposing a truncated live catalog.
[P1] The finalUrl parsing path participates in the authorization-header replay decision; the merge should preserve the patch's fail-closed behavior when the effective response URL cannot be parsed.
[P1] The reviewed branch is behind current main, so exact-head CI and the focused loopback proof should be refreshed after updating the branch.

Maintainer options:

Refresh and land with focused proof (recommended)
Update the branch onto current main, rerun the provider-catalog test and loopback guarded-HTTP proof, and merge if the exact head remains green.
Accept the current SDK behavior change
Maintain the controlled failure for unusable advertised links even though affected nonconforming providers may switch from partial live results to static fallback.

Next step before merge

No automated repair remains; refresh the branch and use normal exact-head merge verification.

Security
Cleared: The patch adds no dependency or execution surface and fails closed when URL parsing cannot establish a safe same-origin credential-replay decision.

Review details

Best possible solution:

Land the fail-closed parsing and incomplete-pagination behavior after refreshing the branch, while preserving same-origin-only credential replay and the cursor fallback for recoverable provider responses.

Do we have a high-confidence way to reproduce the issue?

Yes. The PR provides a high-confidence loopback reproduction through the exported catalog function and production guarded-fetch stack, showing the current-main TypeError and the branch's controlled incomplete result.

Is this the best way to solve the issue?

Yes. Guarding the two external URL parsing sites inside the existing shared catalog runtime, retaining same-origin enforcement, and reporting incomplete pagination is narrower and safer than provider-specific workarounds or silent truncation.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against cc9d80c3d67e.

Label changes

Label changes:

add merge-risk: 🚨 compatibility: The public plugin-SDK helper changes upgrade behavior for providers that return malformed or cross-origin pagination links.
add merge-risk: 🚨 security-boundary: The modified URL parsing also controls whether provider authorization headers may be replayed after guarded-fetch redirects.
add proof: sufficient: Contributor real behavior proof is sufficient. The updated PR body shows after-fix output from the real exported function through a loopback server and the production SSRF-guarded HTTP stack, directly improving on the documented current-main failure.
add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The updated PR body shows after-fix output from the real exported function through a loopback server and the production SSRF-guarded HTTP stack, directly improving on the documented current-main failure.
remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.
remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P2: This is a bounded provider discovery reliability bug that can crash or truncate a live model catalog without affecting the core runtime generally.
merge-risk: 🚨 compatibility: The public plugin-SDK helper changes upgrade behavior for providers that return malformed or cross-origin pagination links.
merge-risk: 🚨 security-boundary: The modified URL parsing also controls whether provider authorization headers may be replayed after guarded-fetch redirects.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The updated PR body shows after-fix output from the real exported function through a loopback server and the production SSRF-guarded HTTP stack, directly improving on the documented current-main failure.
proof: sufficient: Contributor real behavior proof is sufficient. The updated PR body shows after-fix output from the real exported function through a loopback server and the production SSRF-guarded HTTP stack, directly improving on the documented current-main failure.

Evidence reviewed

PR surface:

Source +32, Tests +150. Total +182 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	38	6	+32
Tests	1	150	0	+150
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	188	6	+182

What I checked:

Current-main bug remains: At current main SHA cc9d80c, the pagination resolver constructs URL objects directly from provider response data, so a genuinely malformed next URL can throw before a controlled catalog result is produced. (src/plugin-sdk/provider-catalog-live-runtime.ts:218, cc9d80c3d67e)
Defensive runtime fix: The reviewed head introduces tryParseUrl, retains same-origin pagination, attempts the existing cursor path when an advertised link is unusable, and returns incomplete rather than silently accepting a truncated catalog. (src/plugin-sdk/provider-catalog-live-runtime.ts:215, f09be4cfe322)
Prior finding resolved: Commit f09be4c changes malformed or cross-origin advertised next links without a cursor from an ambiguous completion path to the controlled incomplete result requested in the first review cycle. (src/plugin-sdk/provider-catalog-live-runtime.ts:235, f09be4cfe322)
Focused regression coverage: The updated tests cover malformed top-level and nested next links, cursor recovery, malformed finalUrl handling, and explicit has_more behavior; the PR reports 24 focused tests passing. (src/plugin-sdk/provider-catalog-live-runtime.test.ts:702, f09be4cfe322)
Real guarded-HTTP proof: The revised PR body supplies before/after output from the exported fetchLiveProviderModelIds path using a loopback HTTP server, production fetchWithSsrFGuard, and the production hostname SSRF policy: main throws TypeError: Invalid URL while the branch returns the controlled incomplete-pagination error. (f09be4cfe322)
Public SDK boundary policy: The scoped plugin-SDK policy identifies this directory as a third-party contract and requires narrow provider-family behavior with direct tests; the patch stays within the existing provider-catalog runtime subpath and does not add an export or dependency. (src/plugin-sdk/AGENTS.md:245, cc9d80c3d67e)

Likely related people:

steipete: Repository history attributes the central plugin model-discovery refactor and recent provider runtime work to this account, making it the strongest routing candidate for the shared provider-catalog contract. (role: feature owner and recent provider-runtime contributor; confidence: high; commits: 6745f9a; files: src/plugin-sdk/provider-catalog-live-runtime.ts, src/plugin-sdk/provider-catalog-live-runtime.test.ts)
vincentkoc: Recent merged provider and model-catalog work, including catalog normalization and provider routing changes, makes this account a useful secondary reviewer for live discovery semantics. (role: adjacent provider catalog contributor; confidence: medium; commits: fcb9dcc886e2; files: src/plugin-sdk/provider-catalog-live-runtime.ts, docs/concepts/model-providers.md)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (2 earlier review cycles)

reviewed 2026-07-17T12:41:50.614Z sha fd07ad4 :: needs real behavior proof before merge. :: [P2] Treat malformed next links as incomplete pagination
reviewed 2026-07-17T13:10:22.964Z sha f09be4c :: needs real behavior proof before merge. :: none

…ormed responses Provider catalog live discovery now wraps \ ew URL()\ calls in a \ ryParseUrl\ helper that returns undefined instead of throwing TypeError on malformed absolute URLs from API responses. Before: a malformed \ ext\ / \links.next\ URL or an unparseable fetch redirect target in the catalog response body crashes model discovery with an uncaught TypeError. After: malformed URLs are treated as absent (no next page), falling through to cursor-based pagination or completing the catalog. Co-Authored-By: Claude <[email protected]>

… has no cursor fallback When a provider catalog response advertises a next page via the `next` or `links.next` field but the URL is malformed or cross-origin, the previous fix fell through to bodyAdvertisesMoreLiveModelCatalogPages which could silently return `complete` when has_more was absent or false. This truncated catalogs that use only link-based pagination. The fix returns `incomplete` when a provider-advertised next URL is unusable and no cursor-based fallback exists, surfacing a controlled error instead of silently dropping pages. Co-Authored-By: Claude <[email protected]>

clawsweeper · 2026-07-21T04:44:51Z

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(plugin-sdk): guard provider catalog live URL parsing against malformed responses This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

steipete · 2026-07-21T07:21:56Z

Merged via squash.

Prepared head SHA: 7a45482c9c418a1a47399f217bf678b65193db62
Landed commit: 19fb605fc4f81b81ff12936302c178fe461251b8

* origin/main: (24 commits) fix(agents): keep compaction on live session model (#95713) fix(plugin-sdk): guard provider catalog live URL parsing against malformed responses (#109986) chore(cli): drop dead classifiers and single-use wrappers left by fallback removal (#112191) feat(ui): expand the lobster pet's random universe (#112073) chore(ci): audit dependency fingerprint exports (#112190) fix(ui): preserve graphemes in provider icons (#109509) fix(ui): align settings search with navigation rows (#112172) fix(agents): allow configless gateway rebind to activate standalone owner (#111841) fix(secrets): register secret targets for installed-origin plugins (#104347) improve(ui): show real machine identity in the place picker (#112162) fix: webChat scrollback history is too limited — older assistant replies and tool outputs disappear when scrolling… (#104250) test(ui): run DOM-free suites in Node (#112031) feat(nodes): make computer.act eligibility capability-based for desktop nodes (#112107) fix(apps): harden mobile gateway and watch state fix(ci): restore Z.AI API Platform validation (#112171) fix(ui): prevent Logs controls from overlapping (#112170) fix #95291: message tool fails to deliver files/images on Feishu (400 volc-dcdn / write ECONNRESET) while same Lark SDK upload succeeds standalone (#95514) refactor(cli)!: remove automatic gateway→embedded fallback from openclaw agent (#112074) refactor: move provider transports into packages/ai behind a typed host port (#111669) fix(anthropic): complete transcript reverse-scan windows across short reads (#109431) ...

…ormed responses (openclaw#109986) * fix(plugin-sdk): guard provider catalog live URL parsing against malformed responses Provider catalog live discovery now wraps \ ew URL()\ calls in a \ ryParseUrl\ helper that returns undefined instead of throwing TypeError on malformed absolute URLs from API responses. Before: a malformed \ ext\ / \links.next\ URL or an unparseable fetch redirect target in the catalog response body crashes model discovery with an uncaught TypeError. After: malformed URLs are treated as absent (no next page), falling through to cursor-based pagination or completing the catalog. Co-Authored-By: Claude <[email protected]> * fix(plugin-sdk): return incomplete pagination when malformed next URL has no cursor fallback When a provider catalog response advertises a next page via the `next` or `links.next` field but the URL is malformed or cross-origin, the previous fix fell through to bodyAdvertisesMoreLiveModelCatalogPages which could silently return `complete` when has_more was absent or false. This truncated catalogs that use only link-based pagination. The fix returns `incomplete` when a provider-advertised next URL is unusable and no cursor-based fallback exists, surfacing a controlled error instead of silently dropping pages. Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]>

openclaw-barnacle Bot added the size: S label Jul 17, 2026

steipete self-assigned this Jul 21, 2026

LZY3538 and others added 2 commits July 20, 2026 21:41

steipete force-pushed the fix/catalog-live-url-guard branch from f09be4c to 7a45482 Compare July 21, 2026 04:41

steipete merged commit 19fb605 into openclaw:main Jul 21, 2026
115 checks passed

github-actions Bot mentioned this pull request Jul 21, 2026

📡 Upstream Digest — 2026-07-21 08:10 UTC curtismercier/openclaw-mods#1361

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(plugin-sdk): guard provider catalog live URL parsing against malformed responses#109986

fix(plugin-sdk): guard provider catalog live URL parsing against malformed responses#109986
steipete merged 2 commits into
openclaw:mainfrom
LZY3538:fix/catalog-live-url-guard

LZY3538 commented Jul 17, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 21, 2026

Uh oh!

Uh oh!

steipete commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

LZY3538 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jul 21, 2026

Uh oh!

Uh oh!

steipete commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LZY3538 commented Jul 17, 2026 •

edited

Loading

clawsweeper Bot commented Jul 17, 2026 •

edited

Loading