Skip to content

[plugin sdk] Phase 1: consolidate host workflow seams and media extraction runtime#80229

Closed
100yenadmin wants to merge 45 commits into
openclaw:mainfrom
electricsheephq:codex/plugin-sdk-consolidation
Closed

[plugin sdk] Phase 1: consolidate host workflow seams and media extraction runtime#80229
100yenadmin wants to merge 45 commits into
openclaw:mainfrom
electricsheephq:codex/plugin-sdk-consolidation

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented May 10, 2026

Copy link
Copy Markdown
Contributor

Closes #80219.

This is Phase 1 of the whole-surface Plugin SDK consolidation plan in #80219. It intentionally cleans up the host-workflow/session-control slice plus the active proof-gated seams, not the entire SDK/API surface in one PR.

This PR consolidates the active Plugin SDK host-hook/runtime work into one cleanup-first branch instead of asking reviewers to reason across four separate PRs plus a follow-up architecture issue.

It folds together:

It also adds one narrow consolidation slice on top:

  • additive grouped host-hook aliases on OpenClawPluginApi
  • shared late-callability metadata for runtime-callable post-register APIs

Why this matters

OpenClaw is moving more host behavior into plugins and extensions, but the current Plugin SDK surface was heading toward a flat pile of new top-level verbs. That makes every new capability feel like permanent API expansion even when the underlying runtime seams are valid.

This PR keeps the feature work, but makes the shape easier to grow from:

  • typed session actions stay a clean control/RPC seam
  • session attachments and scheduled turns stay host-mediated workflow seams
  • structured extraction stays inside media understanding instead of widening generic runtime.llm
  • the new grouped aliases make the host-hook-heavy parts of the SDK read like families instead of unrelated top-level escapes

What this PR does

1. Integrates the active host-hook/runtime slices

  • session action gateway protocol, dispatch, and scope enforcement
  • host-mediated session attachment delivery
  • scheduled session turns plus tag-based unscheduling
  • structured extraction in the media-understanding runtime

2. Adds a narrow consolidation layer

  • api.session.state.*
  • api.session.workflow.*
  • api.session.controls.*
  • api.agent.events.*
  • api.runContext.*
  • api.lifecycle.*

These are additive aliases over the current flat methods. Existing plugin code keeps working.

3. Centralizes loader late-callability policy

Instead of loader.ts hardcoding post-register callable methods inline, this PR moves that policy into shared metadata so the runtime-callable host-hook seams have one source of truth.

Current runtime-callable post-register methods remain:

  • sendSessionAttachment
  • scheduleSessionTurn
  • unscheduleSessionTurnsByTag

Architecture

flowchart TD
    A["Plugin author surface"] --> B["OpenClawPluginApi"]
    B --> C["Flat legacy methods remain supported"]
    B --> D["Additive grouped aliases"]

    D --> E["session.state"]
    D --> F["session.workflow"]
    D --> G["session.controls"]
    D --> H["agent.events"]
    D --> I["runContext"]
    D --> J["lifecycle"]

    F --> K["next-turn injection"]
    F --> L["attachments"]
    F --> M["scheduled turns"]
    G --> N["session actions"]
    H --> O["event subscribe/emit"]
    J --> P["runtime cleanup"]

    Q["mediaUnderstanding runtime"] --> R["structured extraction"]
    Q --> S["explicit provider/model media inference lane"]
Loading

Plugin families this supports

This combined surface is meant to support more than one product feature. Concretely, it is a foundation for:

  • approval and operator-control plugins
  • reminder, follow-up, and scheduled workflow plugins
  • channel companion plugins that need outbound attachment delivery
  • session-aware UI/control plugins
  • background monitor and lifecycle plugins
  • media-understanding and structured extraction plugins

Boundaries

  • This does not collapse structured extraction into generic runtime.llm.complete.
  • This does not remove the flat Plugin SDK methods yet.
  • This does not try to genericize all workflow seams into one catch-all runtime mutation API.
  • This keeps the upstream ACP streaming/default-delivery behavior intact.

Real behavior proof

Behavior or issue addressed:
This combined branch keeps the new host-hook seams narrow while proving they work together on the #80229 head. After this patch, typed plugin session actions dispatch through the Gateway with the action-declared operator scope bundle, grouped late-call workflow aliases remain callable after register() closes for bundled-plugin attachment and scheduled-turn operations, registration-only APIs stay blocked after register, and bounded structured extraction stays inside the media-understanding runtime instead of widening generic runtime.llm.

Real environment tested:
Local macOS checkout at /Volumes/LEXAR/repos/openclaw-plugin-sdk-consolidation on head 564400622821b7f89b374f2e7c25ac0de61bfa4d. I used an isolated temp workspace/state/config root at /var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs, started a token-auth loopback Gateway on ws://127.0.0.1:18997, ran live gateway call requests for the session-action seam, ran a standalone bundled-origin guarded API script against that live cron.* backend for grouped workflow aliases, and ran a standalone runtime proof script for extractStructuredWithModel(...).

Exact steps or command run after this patch:

  1. Started an isolated token-auth Gateway from the merged head:
    HOME=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/home \
    OPENCLAW_WORKSPACE_DIR=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/workspace \
    OPENCLAW_STATE_DIR=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/state \
    OPENCLAW_CONFIG_PATH=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/config/openclaw.json \
    pnpm openclaw gateway run --allow-unconfigured --auth token --token proof-token --bind loopback --port 18997
  2. Called the typed session-action seam on that Gateway once with valid payload and once with invalid payload:
    HOME=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/home \
    OPENCLAW_WORKSPACE_DIR=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/workspace \
    OPENCLAW_STATE_DIR=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/state \
    OPENCLAW_CONFIG_PATH=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/config/openclaw.json \
    pnpm openclaw gateway call plugins.sessionAction --json --token proof-token --url ws://127.0.0.1:18997 --params '{"pluginId":"proof-combined","actionId":"approve","sessionKey":"agent:main:main","payload":{"message":"post-fix"}}'
    
    HOME=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/home \
    OPENCLAW_WORKSPACE_DIR=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/workspace \
    OPENCLAW_STATE_DIR=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/state \
    OPENCLAW_CONFIG_PATH=/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/config/openclaw.json \
    pnpm openclaw gateway call plugins.sessionAction --json --token proof-token --url ws://127.0.0.1:18997 --params '{"pluginId":"proof-combined","actionId":"approve","sessionKey":"agent:main:main","payload":{"message":42}}'
  3. Ran a standalone node --import tsx proof script from repo root that:
    • creates a real bundled-origin plugin record through createPluginRegistry(...)
    • captures the guarded API via loader.__testing.runPluginRegisterSync(...)
    • invokes capturedApi.session.workflow.sendSessionAttachment(...)
    • verifies capturedApi.session.state.registerSessionExtension(...) stays blocked after register()
    • invokes capturedApi.session.workflow.scheduleSessionTurn(...) twice plus invalid tag/deleteAfterRun cases
    • calls capturedApi.session.workflow.unscheduleSessionTurnsByTag(...)
    • queries live cron.list before and after cleanup through the running Gateway
  4. Ran a standalone node --import tsx runtime proof script from repo root that invokes runtime.mediaUnderstanding.extractStructuredWithModel(...) once with image+text input and once with text-only input.

Evidence after fix:

$ pnpm openclaw gateway call plugins.sessionAction --json --token proof-token --url ws://127.0.0.1:18997 --params '{"pluginId":"proof-combined","actionId":"approve","sessionKey":"agent:main:main","payload":{"message":"post-fix"}}'
{
  "ok": true,
  "result": {
    "approved": true,
    "message": "post-fix",
    "sessionKey": "agent:main:main",
    "scopes": [
      "operator.admin",
      "operator.read",
      "operator.write",
      "operator.approvals",
      "operator.pairing",
      "operator.talk.secrets"
    ]
  },
  "continueAgent": true
}

$ pnpm openclaw gateway call plugins.sessionAction --json --token proof-token --url ws://127.0.0.1:18997 --params '{"pluginId":"proof-combined","actionId":"approve","sessionKey":"agent:main:main","payload":{"message":42}}'
Gateway call failed: GatewayClientRequestError: plugin session action payload does not match schema: message: must be string

$ node --import tsx <bundled-origin guarded workflow proof>
{
  "attachment": {
    "ok": true,
    "channel": "proofchat",
    "deliveredTo": "12345",
    "count": 1
  },
  "deliveries": [
    {
      "to": "12345",
      "text": "attachment ready",
      "mediaUrl": "/var/folders/l7/tyvn8qfs50v57w__75s3k0f00000gp/T/openclaw-pr80229-proof-qGy5Vs/workspace/proof-report.txt",
      "accountId": "default"
    }
  ],
  "sessionExtensionsBefore": 0,
  "sessionExtensionsAfter": 0,
  "first": {
    "id": "f2f6d165-2356-402c-ab9a-9b67b4e62f31",
    "pluginId": "proof-combined",
    "sessionKey": "agent:main:main",
    "kind": "session-turn"
  },
  "second": {
    "id": "32447765-6139-4238-bc4e-f3fb15ed333d",
    "pluginId": "proof-combined",
    "sessionKey": "agent:main:main",
    "kind": "session-turn"
  },
  "badTag": null,
  "badDelete": null,
  "cronJobsAfterSchedule": {
    "jobs": [
      {
        "id": "32447765-6139-4238-bc4e-f3fb15ed333d",
        "name": "plugin:proof-combined:tag:nudge:agent:main:main:2396fa1c-f78a-49e6-93d0-9e0013a02afa",
        "deleteAfterRun": true,
        "sessionTarget": "session:agent:main:main",
        "payload": { "kind": "agentTurn", "message": "wake two" },
        "delivery": { "mode": "none" }
      },
      {
        "id": "f2f6d165-2356-402c-ab9a-9b67b4e62f31",
        "name": "plugin:proof-combined:tag:nudge:agent:main:main:4e148d8e-5e2b-4cdc-bc80-ce50f6cd8f47",
        "deleteAfterRun": true,
        "sessionTarget": "session:agent:main:main",
        "payload": { "kind": "agentTurn", "message": "wake one" },
        "delivery": { "mode": "announce", "channel": "last" }
      }
    ],
    "total": 2
  },
  "removed": {
    "removed": 2,
    "failed": 0
  },
  "cronJobsAfterCleanup": {
    "jobs": [],
    "total": 0
  }
}

$ node --import tsx <structured extraction runtime proof>
{
  "success": {
    "text": "{\"summary\":\"red square\",\"tags\":[\"shape\"]}",
    "model": "gpt-5.4",
    "provider": "codex",
    "contentType": "json",
    "parsed": {
      "summary": "red square",
      "tags": ["shape"]
    }
  },
  "authProfileIds": ["openai-codex:work"],
  "requestMethods": ["model/list", "thread/start", "turn/start"],
  "threadStart": {
    "approvalPolicy": "on-request",
    "sandbox": "read-only",
    "serviceName": "OpenClaw",
    "dynamicTools": [],
    "ephemeral": true
  },
  "guardError": "Structured extraction requires at least one image input."
}

Observed result after fix:
On the combined #80229 head, plugins.sessionAction dispatch succeeds with typed output and reaches the handler under the action-declared operator scope bundle, while malformed payloads are rejected before handler execution. The grouped late-call workflow aliases on api.session.workflow.* remain callable after register() closes for a bundled plugin, the registration-only session.state.registerSessionExtension(...) path stays blocked (sessionExtensionsBefore/After remained 0), a relative attachment path resolves against the session workspace and is delivered to the active direct-outbound route, two real host cron jobs are created and visible through cron.list, invalid tag/deleteAfterRun inputs fail closed, and tagged cleanup removes both jobs. Structured extraction stays bounded in the media-understanding runtime, forwards the selected auth profile into the provider-owned runtime, returns parsed JSON for image input, and rejects text-only calls before provider dispatch.

What was not tested:
This proof did not run a full end-to-end Control UI or native Apple client flow, and it did not use a credentialed live Codex desktop session. The live evidence here is intentionally limited to merged-head Gateway dispatch, bundled-origin guarded workflow aliases against a real cron.* backend, direct-outbound attachment delivery, and bounded runtime/provider structured extraction behavior.

Verification

Local verification on this combined branch:

node scripts/run-vitest.mjs run --config test/vitest/vitest.config.ts src/plugins/loader.test.ts --maxWorkers=1
node scripts/run-vitest.mjs run --config test/vitest/vitest.contracts-plugin.config.ts src/plugins/contracts/run-context-lifecycle.contract.test.ts src/plugins/contracts/session-actions.contract.test.ts --maxWorkers=1
pnpm plugin-sdk:api:gen
node scripts/run-oxlint-shards.mjs --threads=8

Notes for reviewers

  • The feature work here originated as four active PR lines because each seam was independently useful.
  • The maintainer concern about API accretion is valid, so this branch intentionally adds only the smallest consolidation layer that improves the author-facing shape without re-plumbing the runtime.
  • If this direction lands, the next cleanup step should be export-surface reduction and alias/deprecation of the obvious debt lanes described in [plugin sdk] Consolidate author surface, lifecycle semantics, and export sprawl #80219.

Eva and others added 30 commits May 10, 2026 17:24
Squashed rebase of PR #75581 onto upstream/main (rebase-2026-05-08).
Drops PR's outbound-base.ts and macOS GatewayModels.swift since
upstream already deleted those and integrated the attachment
delivery surface into createTelegramOutboundAdapter.

Original commits in PR include 19 follow-up fixes; squashed for
clean rebase. Refresh plugin-sdk-api-baseline.

Replaces part of #73384/#74483.
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation channel: googlechat Channel integration: googlechat channel: matrix Channel integration: matrix channel: telegram Channel integration: telegram app: web-ui App: web-ui gateway Gateway runtime scripts Repository scripts agents Agent runtime and tooling extensions: codex plugin: file-transfer labels May 10, 2026
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 10, 2026
@clawsweeper

clawsweeper Bot commented May 10, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge.

Summary
The PR adds Plugin SDK session actions, host-mediated session attachments, scheduled session turns, structured media extraction, grouped SDK aliases, protocol/docs/codegen updates, and related regression coverage.

Reproducibility: not applicable. for this PR as a feature/API consolidation rather than an issue report. The review finding is source-reproducible: durable Telegram delivery pre-renders text through the adapter chunker, then the PR makes sendMessageTelegram treat that pre-rendered HTML as Markdown unless an explicit parse mode is present.

Real behavior proof
Needs stronger real behavior proof before merge: The PR body includes live terminal proof for loopback Gateway, proofchat attachment/scheduled-turn, and structured extraction paths, but not live Telegram proof for the touched parse-mode/default-delivery transport behavior. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, ask a maintainer to comment @clawsweeper re-review.

Next step before merge
Contributor-supplied real Telegram proof and maintainer API review are required; automation should not substitute for missing live transport proof on this external PR.

Security
Cleared: Cleared: the sampled diff adds no dependency, workflow, lockfile, or package-lifecycle execution change, and the sensitive new host paths are scope-checked, file-bounded, or read-only runtime mediated.

Review findings

  • [P2] Keep HTML mode for pre-rendered Telegram chunks — extensions/telegram/src/outbound-adapter.ts:81
Review details

Best possible solution:

Keep the PR open, fix the Telegram default formatting path, add redacted real Telegram proof for the transport behavior, then have maintainers decide whether this phase-one SDK/Gateway shape should land as a single consolidation branch.

Do we have a high-confidence way to reproduce the issue?

Not applicable for this PR as a feature/API consolidation rather than an issue report. The review finding is source-reproducible: durable Telegram delivery pre-renders text through the adapter chunker, then the PR makes sendMessageTelegram treat that pre-rendered HTML as Markdown unless an explicit parse mode is present.

Is this the best way to solve the issue?

No: the PR is not ready as written because the Telegram adapter change drops HTML mode on an existing pre-rendered path and the branch lacks required real Telegram proof. A safer fix is to preserve HTML mode for adapter-chunked durable delivery while separately supporting raw direct calls that intentionally rely on Telegram Markdown defaults.

Full review comments:

  • [P2] Keep HTML mode for pre-rendered Telegram chunks — extensions/telegram/src/outbound-adapter.ts:81
    The adapter still advertises markdownToTelegramHtmlChunks, and durable delivery applies that chunker before calling sendText. With this line, normal Telegram sends that do not pass an explicit parse mode omit textMode, so sendMessageTelegram defaults back to Markdown and re-renders or escapes the already-HTML chunks. Keep HTML mode for the chunked durable-delivery path, or split raw direct calls from pre-rendered adapter delivery.
    Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.86

Acceptance criteria:

  • Add or update focused coverage for the durable Telegram delivery path where the adapter chunker pre-renders Markdown to Telegram HTML and no explicit parse mode is passed.
  • Run the narrow Telegram/outbound checks that cover extensions/telegram/src/outbound-adapter.ts plus the shared durable delivery path.
  • Provide redacted live Telegram proof for default Markdown delivery and explicit HTML parse-mode attachment/send behavior; redact private IDs, tokens, phone numbers, endpoints, and account details before posting.

What I checked:

  • Live PR context from provided GitHub data: The PR is open, contributor-authored, not draft, on head 5644006, marked proof supplied, and closes the broader consolidation plan at [plugin sdk] Consolidate author surface, lifecycle semantics, and export sprawl #80219 while folding several closed split PRs. (564400622821)
  • Current main lacks the proposed SDK/Gateway/runtime symbols: A current-main source search found no registerSessionAction, sendSessionAttachment, scheduleSessionTurn, unscheduleSessionTurnsByTag, plugins.sessionAction, or extractStructuredWithModel symbols. (1429f9a181c9)
  • Current main plugin host hook baseline: Current main only exposes plugins.uiDescriptors in the plugin host hook handler, so plugins.sessionAction is new to the PR branch. (src/gateway/server-methods/plugin-host-hooks.ts:10, 1429f9a181c9)
  • PR diff scope: The remote PR diff adds the new Gateway method, Plugin SDK host workflow APIs, structured extraction runtime, Telegram parse-mode/default-delivery changes, generated Swift protocol artifacts, docs, and tests. (564400622821)
  • Current Telegram adapter depends on HTML mode after chunking: Current main's Telegram adapter advertises markdownToTelegramHtmlChunks as its chunker and sets textMode: "html" in base send options, keeping pre-rendered chunks from being rendered as Markdown again. (extensions/telegram/src/outbound-adapter.ts:151, 1429f9a181c9)
  • Durable outbound delivery passes chunked text to sendText: The shared durable delivery path plans text with the adapter chunker, then calls handler.sendText(unit.text, unit.overrides) with the chunked text. (src/infra/outbound/deliver.ts:1383, 1429f9a181c9)

Likely related people:

  • Peter Steinberger: Recent current-main history and blame show broad ownership across the Plugin SDK, gateway, media-understanding, Telegram, and outbound runtime surfaces touched by this PR. (role: recent area contributor; confidence: high; commits: f72a520f7b87, d5841f6412e1, 4862d349257a; files: src/plugins/types.ts, src/plugins/registry.ts, src/gateway/server-methods/plugin-host-hooks.ts)
  • Josh Lehman: Current-main history includes custom context/plugin-system extension work adjacent to the Plugin SDK host surface being expanded here. (role: feature-history contributor; confidence: medium; commits: fee91fefceb4; files: src/plugins, src/plugin-sdk)
  • Gustavo Madeira Santana: Current-main history includes Matrix verification hooks and outbound media-root/session delivery work adjacent to the channel/session delivery paths changed by this PR. (role: adjacent channel/session contributor; confidence: medium; commits: 0f7c40e50885, 9adcccadb1b0; files: extensions/matrix/src/matrix/monitor/handler.test-helpers.ts, src/infra/outbound/message.ts, src/config/sessions/delivery-info.ts)
  • Devin Robison: Current-main history includes Telegram writeback/admin-scope work in the same channel transport area affected by the parse-mode/default-delivery changes. (role: adjacent Telegram contributor; confidence: medium; commits: b7d70ade3b99; files: extensions/telegram)

Remaining risk / open question:

  • The review did not execute the PR branch or a live Telegram account; the blocking formatting issue is source-proven from the current durable-delivery path.
  • The grouped Plugin SDK and Gateway protocol shape still needs maintainer product/API judgment because this is a broad additive public surface.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 1429f9a181c9.

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 10, 2026
@100yenadmin 100yenadmin changed the title [plugin sdk] Consolidate host workflow seams and media extraction runtime [plugin sdk] Phase 1: consolidate host workflow seams and media extraction runtime May 10, 2026
@steipete

Copy link
Copy Markdown
Contributor

Thanks Eva. I carried this forward in #80267 so we can apply maintainer fixups on an OpenClaw branch while preserving your commits/credit.

What changed in the replacement:

  • session action Gateway auth now checks the registered action scope before dispatch
  • grouped SDK facades are required on the injected API shape
  • plugin session action results use the Gateway wire name result consistently
  • structured extraction docs/types call out the image-first contract
  • changelog entries are deduped into the current release section

Closing this PR as superseded by #80267.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui channel: googlechat Channel integration: googlechat channel: matrix Channel integration: matrix channel: telegram Channel integration: telegram docs Improvements or additions to documentation extensions: codex gateway Gateway runtime plugin: file-transfer proof: supplied External PR includes structured after-fix real behavior proof. scripts Repository scripts size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plugin sdk] Consolidate author surface, lifecycle semantics, and export sprawl

3 participants