-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
SDK follow-up: host-owned structured plugin inference beyond media-understanding #80188
Copy link
Copy link
Open
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.ClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.Security boundary, credential, authz, sandbox, or sensitive-data risk.issue-rating: 🌊 off-meta tidepoolIssue quality rating does not apply to this item.Issue quality rating does not apply to this item.
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.ClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.Security boundary, credential, authz, sandbox, or sensitive-data risk.issue-rating: 🌊 off-meta tidepoolIssue quality rating does not apply to this item.Issue quality rating does not apply to this item.
Type
Fields
Priority
None yet
Why this should exist
#79334is the right narrow seam for image-first structured extraction, but the maintainer feedback there surfaced the broader platform need: many plugins need bounded host-owned inference, not raw OAuth credentials and not product-specific core routes.Today plugin authors that need typed model output for knowledge-base enrichment, CRM extraction, finance ingestion, support triage, or import pipelines still face two awkward choices:
A generic host-owned inference surface would keep auth, provider routing, timeouts, and safety in OpenClaw while letting plugins request structured work without ever receiving tokens.
Proposal
Add a general runtime API under
api.runtime.llm, something like:api.runtime.llm.completeStructured(...)That is a better long-term fit than broadening media-understanding further or exposing raw OAuth credentials to plugin code.
Desired behavior
jsonModeandjsonSchemamodeland authprofileExisting building blocks
api.runtime.llm.complete(...)already exists for trusted plugin text completionsapi.runtime.modelAuth.getRuntimeAuthForModel(...)already resolves runtime auth#79334proves a bounded provider-owned structured extraction lane for media inputsThis follow-up would generalize the pattern without requiring each plugin to invent its own bridge.
Example use cases
Boundary
OpenClaw core owns
Plugins own
Non-goals
#79334; that PR should stay the media-understanding seamAcceptance criteria
api.runtime.llm.complete(...)behavior does not regressRelationship to
#79334That PR should remain intentionally narrow: image-first structured extraction via media-understanding.
This issue tracks the broader host-owned plugin inference seam so future plugins do not need to keep stretching media-understanding to cover every structured workload.