Why this matters

OpenClaw plugins increasingly need to turn unstructured user content into safe, typed data: receipts into expense records, screenshots into support evidence, invoices into accounting fields, customer messages into CRM notes, PDFs into knowledge-base snippets, and product photos into searchable inventory metadata.

Today each plugin has to choose between two bad options:

• implement its own model/auth/runtime bridge, usually requiring another user-managed API key; or
• add product-specific extraction routes to core, which does not scale as the plugin ecosystem grows.

This PR adds the missing middle layer: a generic structured extraction capability in the media-understanding SDK. Product plugins keep owning their routes, schemas, storage, and UX, while OpenClaw owns the provider/runtime boundary, auth source, safety posture, and typed SDK contract.

What plugin authors can build with this

Examples this unlocks without adding plugin-specific logic to OpenClaw core:

• Support plugins: extract error messages, stack traces, product names, issue category, severity, and reproduction steps from screenshots.
• Knowledge-base plugins: convert documents or screenshots into normalized metadata and searchable evidence records.
• CRM/sales plugins: extract companies, people, dates, action items, sentiment, and deal updates from inbound media plus short text context.
• Finance/admin plugins: extract vendor, total, currency, tax, due date, and line-item hints from receipts or invoices.
• Inventory/media plugins: extract labels, visible text, tags, object categories, and image summaries from uploaded photos.
• Migration/import plugins: map arbitrary image inputs into a plugin-owned JSON schema before writing to the plugin's own database.

The important part: the plugin defines the schema and decides what to do with the result. OpenClaw only provides the generic, bounded extraction lane.

New SDK shape

This PR adds:

• optional provider method: MediaUnderstandingProvider.extractStructured(...)
• runtime helper: api.runtime.mediaUnderstanding.extractStructuredWithModel(...)
• typed inputs for images plus optional supplemental text context
• optional schemaName, jsonSchema, jsonMode, and timeoutMs
• controlled result metadata: raw text, parsed JSON when JSON mode is enabled, model/provider, and content type

Example plugin call:

const result = await api.runtime.mediaUnderstanding.extractStructuredWithModel({
  provider: "codex",
  model: "gpt-5.5",
  input: [
    {
      type: "image",
      buffer: receiptImageBuffer,
      fileName: "receipt.png",
      mime: "image/png",
    },
    { type: "text", text: "Prefer the printed total over handwritten notes." },
  ],
  instructions: "Extract vendor, total, and searchable tags.",
  schemaName: "receipt.evidence",
  jsonSchema: {
    type: "object",
    properties: {
      vendor: { type: "string" },
      total: { type: "number" },
      tags: { type: "array", items: { type: "string" } },
    },
    required: ["vendor", "total"],
  },
  cfg: api.config,
});

Runtime architecture

flowchart LR
  Plugin["Plugin route, skill, or importer"] --> Runtime["api.runtime.mediaUnderstanding.extractStructuredWithModel"]
  Runtime --> Provider["MediaUnderstandingProvider.extractStructured"]
  Provider --> HostRuntime["Provider-owned host runtime"]
  HostRuntime --> Result["JSON result or controlled error"]
  Result --> Plugin
  Plugin --> Storage["Plugin-owned storage, tools, or user workflow"]

For the bundled Codex provider, this uses the existing Codex app-server/OAuth path rather than requiring a user-supplied model API key.

flowchart LR
  Plugin["Any OpenClaw plugin"] --> SDK["Structured extraction SDK"]
  SDK --> CodexProvider["Codex media-understanding provider"]
  CodexProvider --> AppServer["Codex app-server / OAuth runtime"]
  AppServer --> BoundedTurn["Ephemeral no-tools turn"]
  BoundedTurn --> JSON["Parsed JSON or controlled error"]

Safety and boundaries

The Codex implementation keeps the same bounded posture as image understanding:

• ephemeral thread
• read-only sandbox
• no dynamic tools
• approval policy set to on-request, with approval requests denied by the provider handler
• timeout enforcement
• model modality validation before turn start
• JSON parsing failure returned as a controlled error
• text-only extraction rejected at the runtime seam, keeping this image-first instead of turning it into a generic text completion lane
• no product-specific route names, storage models, or schemas in OpenClaw core

This is intentionally a platform seam, not a feature-specific integration.

What changed

• Adds structured extraction request/result types to the media-understanding SDK.
• Adds extractStructuredWithModel(...) to the plugin runtime media-understanding facade.
• Implements extractStructured(...) in the bundled Codex provider.
• Preserves explicit config-provider image descriptions by keeping describeImageFileWithModel(...) on the full media-understanding registry instead of narrowing it to manifest-only plugin providers.
• Forwards structured extraction auth-profile selection through the runtime helper so provider-owned OAuth/app-server runtimes can honor plugin-selected credentials.
• Narrows the new seam to image-first extraction with optional supplemental text context instead of overlapping general text-only completion surfaces.
• Adds tests for bounded Codex structured extraction, invalid JSON/schema handling, runtime routing, auth-profile forwarding, image-required guardrails, direct image-model registry routing, provider lookup failure, and runtime API exposure.
• Documents the new runtime helper and the plugin/core ownership boundary.
• Adds the required changelog entry for the new plugin SDK/runtime capability.

Relationship to existing LLM surfaces

OpenClaw already has api.runtime.llm.complete for trusted plugin text completions, and llm-task for workflow/tool-level JSON tasks. This PR is narrower and lower-level: a provider SDK/runtime media-understanding seam for schema-shaped extraction over image inputs with optional text context. That keeps extraction provider-owned and plugin-consumable without turning it into a general-purpose arbitrary Codex call API.

Non-goals

• This does not add a product-specific extraction route to OpenClaw core.
• This does not choose any plugin's storage model or JSON schema.
• This does not replace existing image/audio/video media-understanding helpers.
• This does not require plugins to use Codex; other providers can implement the same optional method.
• This does not expand into generic text-only extraction; callers that want arbitrary text completions should keep using the existing LLM surfaces.

Background

This closes #79321.

The immediate downstream need came from a GBrain/OpenClaw integration, but the implementation here is deliberately generic. GBrain, support, CRM, finance, inventory, migration, and knowledge-base plugins can all consume the same SDK seam while keeping their own product-specific routes and schemas outside OpenClaw core.

Real behavior proof

Behavior or issue addressed: The rebased branch exposes a typed plugin-runtime structured extraction seam that dispatches through a registered media-understanding provider, preserves the bounded Codex worker defaults, forwards the selected auth profile into the provider-owned runtime, and rejects text-only calls before provider dispatch.

Real environment tested: Local macOS OpenClaw checkout at /Users/lume/openclaw-review-worktrees/pr-79334-rebase, rebased head 78cfe4a76161fc7d3029beb4edcf7120a94a4d8b, using a standalone node --import tsx proof command outside Vitest. The proof registers the real bundled Codex media-understanding provider in the active plugin runtime registry with a stubbed app-server client, then calls createPluginRuntime().mediaUnderstanding.extractStructuredWithModel(...) once with image-plus-text input and once with text-only input.

Exact steps or command run after this patch:

cd /Users/lume/openclaw-review-worktrees/pr-79334-rebase
node --import tsx <<'EOF'
import { buildCodexMediaUnderstandingProvider } from './extensions/codex/media-understanding-provider.ts';
import { createPluginRuntime } from './src/plugins/runtime/index.ts';
import { createEmptyPluginRegistry } from './src/plugins/registry-empty.ts';
import { resetPluginRuntimeStateForTest, setActivePluginRegistry } from './src/plugins/runtime.ts';

function codexModel(inputModalities = ['text', 'image']) {
  return {
    id: 'gpt-5.4',
    model: 'gpt-5.4',
    upgrade: null,
    upgradeInfo: null,
    availabilityNux: null,
    displayName: 'gpt-5.4',
    description: 'GPT-5.4',
    hidden: false,
    supportedReasoningEfforts: [{ reasoningEffort: 'low', description: 'fast' }],
    defaultReasoningEffort: 'low',
    inputModalities,
    supportsPersonality: false,
    additionalSpeedTiers: [],
    isDefault: true,
  };
}

function threadStartResult() {
  return {
    thread: {
      id: 'thread-1',
      sessionId: 'session-1',
      forkedFromId: null,
      preview: '',
      ephemeral: true,
      modelProvider: 'openai',
      createdAt: 1,
      updatedAt: 1,
      status: { type: 'idle' },
      path: null,
      cwd: process.cwd(),
      cliVersion: '0.125.0',
      source: 'unknown',
      agentNickname: null,
      agentRole: null,
      gitInfo: null,
      name: null,
      turns: [],
    },
    model: 'gpt-5.4',
    modelProvider: 'openai',
    serviceTier: null,
    cwd: process.cwd(),
    instructionSources: [],
    approvalPolicy: 'on-request',
    approvalsReviewer: 'user',
    sandbox: { type: 'dangerFullAccess' },
    permissionProfile: null,
    reasoningEffort: null,
  };
}

function turnStartResult(status = 'inProgress', items = []) {
  return {
    turn: {
      id: 'turn-1',
      status,
      items,
      error: null,
      startedAt: null,
      completedAt: null,
      durationMs: null,
    },
  };
}

function createFakeClient(responseText) {
  const notifications = new Set();
  const requestHandlers = new Set();
  const requests = [];
  const request = async (method, params) => {
    requests.push({ method, params });
    if (method === 'model/list') return { data: [codexModel()], nextCursor: null };
    if (method === 'thread/start') return threadStartResult();
    if (method === 'turn/start') {
      for (const notify of notifications) {
        notify({ method: 'item/agentMessage/delta', params: { threadId: 'thread-1', turnId: 'turn-1', itemId: 'msg-1', delta: responseText } });
        notify({ method: 'turn/completed', params: { threadId: 'thread-1', turnId: 'turn-1', turn: turnStartResult('completed').turn } });
      }
      for (const handler of requestHandlers) handler({ method: 'item/permissions/requestApproval' });
      return turnStartResult();
    }
    return {};
  };
  return {
    client: {
      request,
      addNotificationHandler(handler) { notifications.add(handler); return () => notifications.delete(handler); },
      addRequestHandler(handler) { requestHandlers.add(handler); return () => requestHandlers.delete(handler); },
      close() {},
    },
    requests,
  };
}

const authProfileIds = [];
const { client, requests } = createFakeClient('{"summary":"red square","tags":["shape"]}');
const provider = buildCodexMediaUnderstandingProvider({
  clientFactory: async (_startOptions, authProfileId) => {
    authProfileIds.push(authProfileId ?? null);
    return client;
  },
});

const registry = createEmptyPluginRegistry();
registry.mediaUnderstandingProviders.push({
  pluginId: 'codex',
  pluginName: 'Codex',
  source: 'proof-script',
  provider,
});
setActivePluginRegistry(registry, 'proof-script', 'default', process.cwd());

const runtime = createPluginRuntime();
const success = await runtime.mediaUnderstanding.extractStructuredWithModel({
  provider: 'codex',
  model: 'gpt-5.4',
  input: [
    {
      type: 'image',
      buffer: Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+kX3sAAAAASUVORK5CYII=', 'base64'),
      fileName: 'red-square.png',
      mime: 'image/png',
    },
    { type: 'text', text: 'Return searchable evidence for the uploaded image.' },
  ],
  instructions: 'Return JSON with summary and tags.',
  schemaName: 'proof.red-square',
  jsonSchema: {
    type: 'object',
    properties: {
      summary: { type: 'string' },
      tags: { type: 'array', items: { type: 'string' } },
    },
    required: ['summary'],
  },
  profile: 'openai-codex:work',
  cfg: {},
  agentDir: process.cwd(),
});

let guardError = null;
try {
  await runtime.mediaUnderstanding.extractStructuredWithModel({
    provider: 'codex',
    model: 'gpt-5.4',
    input: [{ type: 'text', text: 'No image present.' }],
    instructions: 'Return JSON.',
    cfg: {},
    agentDir: process.cwd(),
  });
} catch (error) {
  guardError = error instanceof Error ? error.message : String(error);
}

console.log(JSON.stringify({
  success,
  authProfileIds,
  requestMethods: requests.map((entry) => entry.method),
  threadStart: requests.find((entry) => entry.method === 'thread/start')?.params,
  turnInput: requests.find((entry) => entry.method === 'turn/start')?.params?.input,
  guardError,
}, null, 2));

resetPluginRuntimeStateForTest();
EOF

Evidence after fix:

{
  "success": {
    "text": "{\"summary\":\"red square\",\"tags\":[\"shape\"]}",
    "model": "gpt-5.4",
    "provider": "codex",
    "contentType": "json",
    "parsed": {
      "summary": "red square",
      "tags": [
        "shape"
      ]
    }
  },
  "authProfileIds": [
    "openai-codex:work"
  ],
  "requestMethods": [
    "model/list",
    "thread/start",
    "turn/start"
  ],
  "threadStart": {
    "model": "gpt-5.4",
    "modelProvider": "openai",
    "cwd": "/Users/lume/openclaw-review-worktrees/pr-79334-rebase",
    "approvalPolicy": "on-request",
    "sandbox": "read-only",
    "serviceName": "OpenClaw",
    "developerInstructions": "You are OpenClaw's bounded structured-extraction worker. Return only the requested extraction. Do not call tools, edit files, ask follow-up questions, or include secrets.",
    "dynamicTools": [],
    "experimentalRawEvents": true,
    "persistExtendedHistory": false,
    "ephemeral": true
  },
  "turnInput": [
    {
      "type": "text",
      "text": "Return JSON with summary and tags.\n\nSchema name: proof.red-square\n\nJSON schema:\n{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\"},\"tags\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}},\"required\":[\"summary\"]}\n\nReturn valid JSON only. Do not wrap the JSON in Markdown fences.",
      "text_elements": []
    },
    {
      "type": "image",
      "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+kX3sAAAAASUVORK5CYII="
    },
    {
      "type": "text",
      "text": "Return searchable evidence for the uploaded image.",
      "text_elements": []
    }
  ],
  "guardError": "Structured extraction requires at least one image input."
}

Observed result after fix: The plugin runtime facade dispatched extractStructuredWithModel(...) through the registered Codex media-understanding provider, the provider returned parsed JSON on the bounded app-server path, the selected auth profile reached the provider-owned runtime, and the text-only call failed early with the intended image-required guard instead of widening this seam into general text extraction.

What was not tested: This proof intentionally uses a stubbed app-server client so it can exercise the real runtime/provider dispatch path deterministically in a local checkout without requiring a desktop-bound live OAuth session. The PR does not include a credentialed live Codex desktop turn artifact because that would require shipping private local auth/session material into public review evidence.

Validation

• pnpm install --frozen-lockfile
• pnpm plugin-sdk:api:gen
• pnpm plugin-sdk:api:check
• pnpm test src/media-understanding/runtime.test.ts src/media-understanding/provider-registry.test.ts extensions/codex/media-understanding-provider.test.ts src/plugins/runtime/index.test.ts
• pnpm check:changed