Why this should exist

OpenClaw already has two useful but incomplete lanes for plugin-side model work:

• api.runtime.llm.complete(...) for trusted host-owned text completion
• api.runtime.mediaUnderstanding.extractStructuredWithModel(...) in #79334 for provider-owned media-first structured extraction

What is still missing is the general middle lane for plugins that need bounded host-owned structured inference without:

• handling OAuth/API credentials directly,
• inventing bespoke shell bridges, or
• stretching media-understanding into every structured workload.

That gap shows up across a lot more than one plugin family:

• knowledge-base plugins turning text or screenshots into normalized evidence
• support plugins extracting issue summaries, repro steps, and error clues from screenshots plus notes
• CRM plugins extracting companies, people, dates, action items, and sentiment from inbound content
• finance/admin plugins extracting vendor, totals, tax, and due dates from receipts or invoices
• migration/import plugins mapping arbitrary raw content into plugin-owned JSON before storage

The goal of this PR is to make that generic host-owned lane first-class.

What this adds

This PR adds:

• api.runtime.llm.completeStructured(...)
• typed structured input blocks for text and optional images
• optional jsonMode, jsonSchema, schemaName, timeoutMs, and profile
• parsed JSON results when JSON mode is requested
• the same host-owned model/auth/runtime preparation path as api.runtime.llm.complete(...)
• the same trust gating model for model and agent overrides, plus a new explicit gate for auth-profile overrides

Why this belongs under runtime.llm

This API is deliberately the generic agent-bound runtime lane.

It reuses the same host-owned completion path as complete(...), then adds the structured affordances plugin authors actually need:

• prompt shaping
• optional image inputs
• JSON/schema validation
• timeout control
• controlled typed output

That makes it the right home for general structured plugin inference.

Boundary vs #79334

This PR is a sister, not a replacement.

• runtime.llm.completeStructured(...) is the generic agent-bound structured inference lane
• mediaUnderstanding.extractStructuredWithModel(...) in #79334 remains the narrower provider-owned media capability lane

That separation keeps both seams honest:

• use runtime.llm.completeStructured(...) when the plugin wants host-owned structured inference against the active agent/runtime path
• use mediaUnderstanding.extractStructuredWithModel(...) when the plugin wants explicit provider/model media-routing behavior

flowchart LR
  Plugin["Trusted plugin"] --> LLM["api.runtime.llm.completeStructured(...)\nagent-bound generic lane"]
  Plugin --> Media["api.runtime.mediaUnderstanding.extractStructuredWithModel(...)\nprovider-owned media lane"]
  LLM --> Host["host-owned simple completion runtime"]
  Media --> Provider["media-understanding provider registry"]

Safety and trust model

This stays host-owned end to end.

Plugins do not receive raw OAuth tokens, refresh tokens, or provider secrets.

The host still owns:

• auth resolution
• provider runtime preparation
• model routing
• timeout handling
• agent binding
• trust gating

This PR also closes a subtle trust gap by treating auth-profile selection as a real override. profile now requires explicit opt-in via:

• plugins.entries.<id>.llm.allowProfileOverride: true

The full runtime trust picture is now:

• model overrides: allowModelOverride + optional allowedModels
• cross-agent calls: allowAgentIdOverride
• auth-profile selection: allowProfileOverride

Embedded model-ref suffixes now flow through that same gate, and conflicting profile vs model@profile inputs fail closed.

Behavior details

A few implementation details here are deliberate and worth calling out:

• JSON mode is explicit: parsed JSON is only returned when jsonMode: true or jsonSchema is provided
• Image inputs respect host image fallback behavior: when the active model is text-only, structured image calls reuse the host image-model fallback path instead of failing early when a configured image model exists
• Context-engine parity is preserved: session-bound context-engine runtime hooks now expose completeStructured(...) alongside complete(...)

Example

const result = await api.runtime.llm.completeStructured({
  instructions: "Extract vendor, total, and searchable tags.",
  input: [
    {
      type: "image",
      buffer: receiptBuffer,
      mimeType: "image/png",
      fileName: "receipt.png",
    },
    { type: "text", text: "Prefer the printed total over handwritten notes." },
  ],
  jsonSchema: {
    type: "object",
    properties: {
      vendor: { type: "string" },
      total: { type: "number" },
      tags: { type: "array", items: { type: "string" } },
    },
    required: ["vendor", "total"],
  },
  purpose: "receipts.extract",
});

What changed

• add LlmCompleteStructured* runtime types
• add runtime.llm.completeStructured(...) to the plugin runtime facade
• add trust gating for auth-profile override selection
• extend the plugin config schema/docs with plugins.entries.<id>.llm.allowProfileOverride
• reuse host image-model fallback routing for structured image calls
• expose completeStructured(...) to context-engine runtime hooks
• add runtime tests for JSON/schema validation, image inputs, profile trust, image fallback behavior, timeout behavior, and context-engine binding
• add regression tests that block model@profile bypasses and reject conflicting explicit vs embedded auth-profile selection
• regenerate the plugin SDK API baseline

Validation

• pnpm test -- src/plugins/runtime/runtime-llm.runtime.test.ts src/plugins/runtime/index.test.ts src/plugins/config-state.test.ts src/config/schema.help.quality.test.ts
• pnpm plugin-sdk:api:gen
• pnpm plugin-sdk:api:check
• pnpm check:changed

Real behavior proof

• Behavior or issue addressed:

  • Plugins could express auth-profile selection either as profile: "..." or as a trailing model-ref suffix like openai/gpt-5.5@work.
  • Before this fix, the new trust gate only checked the explicit profile field, so embedded model@profile requests could steer credential selection without allowProfileOverride.

• Real environment tested:

  • Local OpenClaw checkout on codex/plugin-inference-followup using the real runtime implementation through node --import tsx on macOS, with no test mocks in the proof command.

• Exact steps or command run after this patch:

  node --import tsx -e 'import { createRuntimeLlm } from "./src/plugins/runtime/runtime-llm.runtime.ts"; const cfg = { agents: { defaults: { model: "openai/gpt-5.5" } } }; const llm = createRuntimeLlm({ getConfig: () => cfg, authority: { caller: { kind: "host", id: "proof" }, allowComplete: true, allowModelOverride: true, agentId: "ada" } }); for (const [label, params] of [["embedded-profile-blocked", { model: "openai/gpt-5.5@openai-codex:work", instructions: "Extract summary.", input: [{ type: "text", text: "Hello" }] }],["conflicting-profile-rejected", { model: "openai/gpt-5.5@openai-codex:work", profile: "openai-codex:other", instructions: "Extract summary.", input: [{ type: "text", text: "Hello" }] }]]) { try { await llm.completeStructured(params); console.log(label + ": UNEXPECTED_SUCCESS"); } catch (error) { console.log(label + ": " + (error instanceof Error ? error.message : String(error))); } }'

• Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
  Terminal capture:

  embedded-profile-blocked: Plugin LLM completion cannot override the auth profile.
  conflicting-profile-rejected: Plugin LLM completion received conflicting auth profiles in model and profile fields.
  

  Supplemental focused regression coverage:

  ✓ rejects structured auth-profile overrides without explicit trust
  ✓ rejects auth-profile suffixes in structured model refs without explicit trust
  ✓ treats auth-profile suffixes in structured model refs as profile overrides when trusted
  ✓ rejects conflicting explicit and embedded structured auth-profile overrides
  

• Observed result after fix:

  • The real runtime now fails closed on both bypass shapes before host auth preparation.
  • Structured callers can no longer pick a credential profile through model@profile unless allowProfileOverride is explicitly trusted.
  • Conflicting explicit and embedded profile selections are rejected instead of being resolved implicitly.

• What was not tested:

  • A live provider-backed completion after an allowed profile override was not exercised in this proof step.
  • The trusted-path wiring is covered by the focused runtime tests above.

• Before evidence (optional but encouraged):

  • Before this patch, profile was gated but the embedded model@profile form was not normalized before auth-profile policy evaluation.

Non-goals

• no raw OAuth/token exposure to plugins
• no plugin-specific routes or schemas in core
• no replacement of #79334
• no tool-using long-running agent workflow lane

Closes #80188.