Skip to content

Commit 5aad844

Browse files
steipetesteipete
committed
docs(plugin-sdk): document consolidated workflow seams
1 parent 4f2cb1c commit 5aad844

6 files changed

Lines changed: 114 additions & 15 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ Docs: https://docs.openclaw.ai
3737
- Agents/Codex: remove the configurable Codex dynamic-tools profile so Codex app-server always owns workspace, edit, patch, exec, process, and plan tools while OpenClaw integration tools remain available.
3838
- macOS app: update the Peekaboo bridge dependency to Peekaboo 3.0.0.
3939
- Dependencies: refresh workspace pins and move the WhatsApp plugin from `@whiskeysockets/baileys` to `baileys` while keeping the `7.0.0-rc10` runtime.
40+
- Plugin SDK: add bundled-plugin session actions, `sendSessionAttachment`, and Cron-backed `scheduleSessionTurn`/tag cleanup under the grouped session namespace. Replaces #75578/#75581/#75588 and part of #73384/#74483. Thanks @100yenadmin.
41+
- Plugin SDK/media-understanding: add `extractStructuredWithModel(...)` plus the optional provider-side `extractStructured(...)` seam so trusted plugins can run bounded image-first structured extraction with optional supplemental text context through provider-owned runtimes such as Codex.
4042

4143
### Fixes
4244

@@ -330,6 +332,7 @@ Docs: https://docs.openclaw.ai
330332
- QA/Mantis: accept Blacksmith Testbox `tbx_...` lease ids from desktop smoke warmup, so provider overrides do not fail before inspect/run. Thanks @vincentkoc.
331333
- Plugins/SDK: add bounded `before_agent_finalize` retry instructions so workflow plugins can request one more model pass. Thanks @100yenadmin.
332334
- Plugin SDK: add plugin-owned `SessionEntry` slot projection and scoped trusted-policy session extension reads. (#75609; replaces part of #73384/#74483) Thanks @100yenadmin.
335+
- Plugin SDK/Gateway: add scoped `plugins.sessionAction` dispatch and plugin-attributed `emitAgentEvent` support so plugins can expose typed session actions and workflow events to trusted clients. (#75578; replaces part of #73384/#74483) Thanks @100yenadmin.
333336
- Plugins/SDK: expose host-derived tool target paths to `before_tool_call` and trusted policy hooks so workflow plugins can reason about known file targets without reparsing tool envelopes. (#75605) Thanks @100yenadmin.
334337
- Control UI/WebChat: show a persistent compact context usage indicator from fresh session token data before the high-pressure warning state, while keeping the existing compaction prompt threshold. Fixes #46398; refs #45048, #50071, and #73744. Thanks @walterwkchoy, @AxelrodAI, @Brissux, @vincentkoc, and @BunsDev.
335338
- Contributor PRs: require external pull requests to include after-fix real behavior proof from a real OpenClaw setup, with terminal screenshots, console output, redacted runtime logs, linked artifacts, and copied live output treated as valid evidence while unit tests, mocks, lint, typechecks, snapshots, and CI remain supplemental only.

‎docs/.generated/plugin-sdk-api-baseline.sha256‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
c1d16721a00a26fc578878e508e9a4e2190fe2fadd80534873d0b47ef5e4b513 plugin-sdk-api-baseline.json
2-
cb008db0486c3ba433bfda844a1c2526bb9729afa21f865e0e86f3ad95ac7cfe plugin-sdk-api-baseline.jsonl
1+
19455aee06dd33e2679cfcd8075b10cce806069667097fd7e717aa641c262e51 plugin-sdk-api-baseline.json
2+
ea6e0b36ab14977bed8dcf64118e58a8e58a76f41860c32055a73bcd04612826 plugin-sdk-api-baseline.jsonl

‎docs/plugins/architecture-internals.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -499,6 +499,30 @@ const video = await api.runtime.mediaUnderstanding.describeVideoFile({
499499
filePath: "/tmp/inbound-video.mp4",
500500
cfg: api.config,
501501
});
502+
503+
const extraction = await api.runtime.mediaUnderstanding.extractStructuredWithModel({
504+
provider: "codex",
505+
model: "gpt-5.5",
506+
input: [
507+
{
508+
type: "image",
509+
buffer: receiptImageBuffer,
510+
fileName: "receipt.png",
511+
mime: "image/png",
512+
},
513+
{ type: "text", text: "Use the printed fields as the source of truth." },
514+
],
515+
instructions: "Return entities and searchable tags.",
516+
schemaName: "example.evidence",
517+
jsonSchema: {
518+
type: "object",
519+
properties: {
520+
entities: { type: "array", items: { type: "string" } },
521+
tags: { type: "array", items: { type: "string" } },
522+
},
523+
},
524+
cfg: api.config,
525+
});
502526
```
503527

504528
For audio transcription, plugins can use either the media-understanding runtime
@@ -517,6 +541,11 @@ Notes:
517541

518542
- `api.runtime.mediaUnderstanding.*` is the preferred shared surface for
519543
image/audio/video understanding.
544+
- `extractStructuredWithModel(...)` is the plugin-facing seam for bounded
545+
provider-owned image-first extraction. Include at least one image input;
546+
text inputs are supplemental context.
547+
product plugins own their routes and schemas while OpenClaw owns the
548+
provider/runtime boundary.
520549
- Uses core media-understanding audio configuration (`tools.media.audio`) and provider fallback order.
521550
- Returns `{ text: undefined }` when no transcription output is produced (for example skipped/unsupported input).
522551
- `api.runtime.stt.transcribeAudioFile(...)` remains as a compatibility alias.

‎docs/plugins/sdk-overview.md‎

Lines changed: 46 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -140,18 +140,52 @@ generic contracts; Plan Mode can use them, but so can approval workflows,
140140
workspace policy gates, background monitors, setup wizards, and UI companion
141141
plugins.
142142

143-
| Method | Contract it owns |
144-
| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
145-
| `api.registerSessionExtension(...)` | Plugin-owned, JSON-compatible session state projected through Gateway sessions |
146-
| `api.enqueueNextTurnInjection(...)` | Durable exactly-once context injected into the next agent turn for one session |
147-
| `api.registerTrustedToolPolicy(...)` | Bundled/trusted pre-plugin tool policy that can block or rewrite tool params |
148-
| `api.registerToolMetadata(...)` | Tool catalog display metadata without changing the tool implementation |
149-
| `api.registerCommand(...)` | Scoped plugin commands; command results can set `continueAgent: true`; Discord native commands support `descriptionLocalizations` |
150-
| `api.registerControlUiDescriptor(...)` | Control UI contribution descriptors for session, tool, run, or settings surfaces |
151-
| `api.registerRuntimeLifecycle(...)` | Cleanup callbacks for plugin-owned runtime resources on reset/delete/reload paths |
152-
| `api.registerAgentEventSubscription(...)` | Sanitized event subscriptions for workflow state and monitors |
153-
| `api.setRunContext(...)` / `getRunContext(...)` / `clearRunContext(...)` | Per-run plugin scratch state cleared on terminal run lifecycle |
154-
| `api.registerSessionSchedulerJob(...)` | Plugin-owned session scheduler job records with deterministic cleanup |
143+
| Method | Contract it owns |
144+
| ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
145+
| `api.session.state.registerSessionExtension(...)` | Plugin-owned, JSON-compatible session state projected through Gateway sessions |
146+
| `api.session.workflow.enqueueNextTurnInjection(...)` | Durable exactly-once context injected into the next agent turn for one session |
147+
| `api.registerTrustedToolPolicy(...)` | Bundled/trusted pre-plugin tool policy that can block or rewrite tool params |
148+
| `api.registerToolMetadata(...)` | Tool catalog display metadata without changing the tool implementation |
149+
| `api.registerCommand(...)` | Scoped plugin commands; command results can set `continueAgent: true`; Discord native commands support `descriptionLocalizations` |
150+
| `api.session.controls.registerControlUiDescriptor(...)` | Control UI contribution descriptors for session, tool, run, or settings surfaces |
151+
| `api.lifecycle.registerRuntimeLifecycle(...)` | Cleanup callbacks for plugin-owned runtime resources on reset/delete/reload paths |
152+
| `api.agent.events.registerAgentEventSubscription(...)` | Sanitized event subscriptions for workflow state and monitors |
153+
| `api.runContext.setRunContext(...)` / `getRunContext(...)` / `clearRunContext(...)` | Per-run plugin scratch state cleared on terminal run lifecycle |
154+
| `api.session.workflow.registerSessionSchedulerJob(...)` | Cleanup metadata for plugin-owned scheduler jobs; does not schedule work or create task records |
155+
| `api.session.workflow.sendSessionAttachment(...)` | Bundled-only host-mediated file attachment delivery to the active direct-outbound session route |
156+
| `api.session.workflow.scheduleSessionTurn(...)` / `unscheduleSessionTurnsByTag(...)` | Bundled-only Cron-backed scheduled session turns plus tag-based cleanup |
157+
| `api.session.controls.registerSessionAction(...)` | Typed session actions clients can dispatch through the Gateway |
158+
159+
Use the grouped namespaces for new plugin code:
160+
161+
- `api.session.state.registerSessionExtension(...)`
162+
- `api.session.workflow.enqueueNextTurnInjection(...)`
163+
- `api.session.workflow.registerSessionSchedulerJob(...)`
164+
- `api.session.workflow.sendSessionAttachment(...)`
165+
- `api.session.workflow.scheduleSessionTurn(...)`
166+
- `api.session.workflow.unscheduleSessionTurnsByTag(...)`
167+
- `api.session.controls.registerSessionAction(...)`
168+
- `api.session.controls.registerControlUiDescriptor(...)`
169+
- `api.agent.events.registerAgentEventSubscription(...)`
170+
- `api.agent.events.emitAgentEvent(...)`
171+
- `api.runContext.setRunContext(...)` / `getRunContext(...)` / `clearRunContext(...)`
172+
- `api.lifecycle.registerRuntimeLifecycle(...)`
173+
174+
The equivalent flat methods remain available as deprecated compatibility
175+
aliases for existing plugins. Do not add new plugin code that calls
176+
`api.registerSessionExtension`, `api.enqueueNextTurnInjection`,
177+
`api.registerControlUiDescriptor`, `api.registerRuntimeLifecycle`,
178+
`api.registerAgentEventSubscription`, `api.emitAgentEvent`,
179+
`api.setRunContext`, `api.getRunContext`, `api.clearRunContext`,
180+
`api.registerSessionSchedulerJob`, `api.registerSessionAction`,
181+
`api.sendSessionAttachment`, `api.scheduleSessionTurn`, or
182+
`api.unscheduleSessionTurnsByTag` directly.
183+
184+
`scheduleSessionTurn(...)` is a session-scoped convenience over the Gateway
185+
Cron scheduler. Cron owns timing and creates the background task record when the
186+
turn runs; the Plugin SDK only constrains the target session, plugin-owned
187+
naming, and cleanup. Use `api.runtime.tasks.managedFlows` inside the scheduled
188+
turn when the work itself needs durable multi-step Task Flow state.
155189

156190
The contracts intentionally split authority:
157191

‎docs/plugins/sdk-runtime.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,11 @@ Provider and channel execution paths must use the active runtime config snapshot
217217
<Accordion title="api.runtime.tasks.managedFlows">
218218
Bind a Task Flow runtime to an existing OpenClaw session key or trusted tool context, then create and manage Task Flows without passing an owner on every call.
219219

220+
Task Flow tracks durable multi-step workflow state. It is not a scheduler:
221+
use Cron or `api.session.workflow.scheduleSessionTurn(...)` for future
222+
wakeups, then use `managedFlows` from the scheduled turn when that work
223+
needs flow state, child tasks, waits, or cancellation.
224+
220225
```typescript
221226
const taskFlow = api.runtime.tasks.managedFlows.fromToolContext(ctx);
222227

@@ -300,6 +305,34 @@ Provider and channel execution paths must use the active runtime config snapshot
300305
filePath: "/tmp/inbound-file.pdf",
301306
cfg: api.config,
302307
});
308+
309+
// Structured image extraction through a specific provider/model.
310+
// Include at least one image; text inputs are supplemental context.
311+
const evidence = await api.runtime.mediaUnderstanding.extractStructuredWithModel({
312+
provider: "codex",
313+
model: "gpt-5.5",
314+
input: [
315+
{
316+
type: "image",
317+
buffer: receiptImageBuffer,
318+
fileName: "receipt.png",
319+
mime: "image/png",
320+
},
321+
{ type: "text", text: "Prefer the printed total over handwritten notes." },
322+
],
323+
instructions: "Extract vendor, total, and searchable tags.",
324+
schemaName: "receipt.evidence",
325+
jsonSchema: {
326+
type: "object",
327+
properties: {
328+
vendor: { type: "string" },
329+
total: { type: "number" },
330+
tags: { type: "array", items: { type: "string" } },
331+
},
332+
required: ["vendor", "total"],
333+
},
334+
cfg: api.config,
335+
});
303336
```
304337

305338
Returns `{ text: undefined }` when no output is produced (e.g. skipped input).

‎docs/plugins/sdk-subpaths.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,7 @@ focused channel/runtime subpaths, `config-contracts`, `string-coerce-runtime`,
306306
| `plugin-sdk/media-mime` | Narrow MIME normalization, file-extension mapping, MIME detection, and media-kind helpers |
307307
| `plugin-sdk/media-store` | Narrow media store helpers such as `saveMediaBuffer` |
308308
| `plugin-sdk/media-generation-runtime` | Shared media-generation failover helpers, candidate selection, and missing-model messaging |
309-
| `plugin-sdk/media-understanding` | Media understanding provider types plus provider-facing image/audio helper exports |
309+
| `plugin-sdk/media-understanding` | Media understanding provider types plus provider-facing image/audio/structured-extraction helper exports |
310310
| `plugin-sdk/text-chunking` | Text and markdown chunking/render helpers, markdown table conversion, directive-tag stripping, and safe-text utilities |
311311
| `plugin-sdk/text-chunking` | Outbound text chunking helper |
312312
| `plugin-sdk/speech` | Speech provider types plus provider-facing directive, registry, validation, OpenAI-compatible TTS builder, and speech helper exports |

0 commit comments

Comments
 (0)