Skip to content

SDK follow-up: host-owned structured plugin inference beyond media-understanding #80188

Description

@100yenadmin

Why this should exist

#79334 is the right narrow seam for image-first structured extraction, but the maintainer feedback there surfaced the broader platform need: many plugins need bounded host-owned inference, not raw OAuth credentials and not product-specific core routes.

Today plugin authors that need typed model output for knowledge-base enrichment, CRM extraction, finance ingestion, support triage, or import pipelines still face two awkward choices:

  • ship bespoke bridges that shell out into host runtime behavior
  • ask for product-specific SDK seams one plugin shape at a time

A generic host-owned inference surface would keep auth, provider routing, timeouts, and safety in OpenClaw while letting plugins request structured work without ever receiving tokens.

Proposal

Add a general runtime API under api.runtime.llm, something like:

  • api.runtime.llm.completeStructured(...)

That is a better long-term fit than broadening media-understanding further or exposing raw OAuth credentials to plugin code.

Desired behavior

  • text-only structured completion
  • image-plus-text structured completion
  • optional jsonMode and jsonSchema
  • host-controlled model and auth profile
  • bounded timeout
  • controlled success and error envelopes
  • provider and model metadata in the response

Existing building blocks

  • api.runtime.llm.complete(...) already exists for trusted plugin text completions
  • api.runtime.modelAuth.getRuntimeAuthForModel(...) already resolves runtime auth
  • #79334 proves a bounded provider-owned structured extraction lane for media inputs

This follow-up would generalize the pattern without requiring each plugin to invent its own bridge.

Example use cases

  • KB plugins: turn text or screenshots into normalized evidence records
  • CRM plugins: extract people, companies, dates, action items, and summaries from text plus images
  • Finance plugins: receipt and invoice field extraction
  • Support plugins: screenshot plus reproduction-note triage
  • Migration/import plugins: map raw content into plugin-owned JSON before storage

Boundary

OpenClaw core owns

  • auth resolution
  • provider runtime exchange
  • model/profile policy
  • bounded execution
  • typed generic inference API

Plugins own

  • routes
  • prompts
  • schemas
  • storage
  • follow-on behavior

Non-goals

  • no raw OAuth token or refresh token exposure to plugins
  • no product-specific routes or schemas in core
  • no tool-using long-running agent lane
  • no replacement of #79334; that PR should stay the media-understanding seam

Acceptance criteria

  • no user model API key required when the host runtime already has auth
  • text-only structured requests work
  • image-plus-text structured requests work when the chosen provider supports them
  • JSON/schema validation failures return controlled errors
  • host policy still gates model/profile/agent overrides
  • secrets are not returned or logged
  • existing api.runtime.llm.complete(...) behavior does not regress

Relationship to #79334

That PR should remain intentionally narrow: image-first structured extraction via media-understanding.
This issue tracks the broader host-owned plugin inference seam so future plugins do not need to keep stretching media-understanding to cover every structured workload.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.issue-rating: 🌊 off-meta tidepoolIssue quality rating does not apply to this item.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions