Extensions: add adaptive-cards extension for native GenUI#33486
Extensions: add adaptive-cards extension for native GenUI#33486VikrantSingh01 wants to merge 5 commits intoopenclaw:mainfrom
Conversation
Greptile SummaryThis PR adds an
Confidence Score: 3/5
Last reviewed commit: 9c7451c |
| @@ -0,0 +1,208 @@ | |||
| import { Type } from "@sinclair/typebox"; | |||
| import type { OpenClawPluginApi } from "openclaw/plugin-sdk"; | |||
| import { stringEnum } from "openclaw/plugin-sdk"; | |||
There was a problem hiding this comment.
Unused import
stringEnum is imported but never referenced anywhere in the file. This will likely trigger an oxlint error (no-unused-vars / no-unused-imports) during the pre-commit hook.
| import { stringEnum } from "openclaw/plugin-sdk"; | |
| import type { OpenClawPluginApi } from "openclaw/plugin-sdk"; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/adaptive-cards/index.ts
Line: 3
Comment:
**Unused import**
`stringEnum` is imported but never referenced anywhere in the file. This will likely trigger an oxlint error (`no-unused-vars` / `no-unused-imports`) during the pre-commit hook.
```suggestion
import type { OpenClawPluginApi } from "openclaw/plugin-sdk";
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 38696298. Removed the unused stringEnum import.
| const markedText = [ | ||
| fallback, | ||
| "", | ||
| `${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`, | ||
| templateData | ||
| ? `<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->` | ||
| : "", | ||
| ] | ||
| .filter(Boolean) | ||
| .join("\n"); |
There was a problem hiding this comment.
filter(Boolean) silently drops the blank-line separator
The empty string "" on line 96 is intended to produce a blank line between the fallback text and the card marker. However, .filter(Boolean) removes all falsy values — including that empty string — so the blank line is never emitted. When fallback is non-empty the output becomes "<fallback>\n<!--adaptive-card-->..." (single newline, no blank line). When fallback is empty (e.g., the card body contains only Image or Input.* elements that generateFallbackText doesn't handle), the output is just the raw <!--adaptive-card--> markers with no human-readable text at all — meaning channels that don't parse markers (Telegram, IRC, Slack) show nothing or raw comment syntax.
A more explicit construction avoids both problems:
| const markedText = [ | |
| fallback, | |
| "", | |
| `${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`, | |
| templateData | |
| ? `<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->` | |
| : "", | |
| ] | |
| .filter(Boolean) | |
| .join("\n"); | |
| const parts: string[] = []; | |
| if (fallback) { | |
| parts.push(fallback, ""); | |
| } | |
| parts.push(`${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`); | |
| if (templateData) { | |
| parts.push( | |
| `<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->`, | |
| ); | |
| } | |
| const markedText = parts.join("\n"); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/adaptive-cards/index.ts
Line: 95-104
Comment:
**`filter(Boolean)` silently drops the blank-line separator**
The empty string `""` on line 96 is intended to produce a blank line between the fallback text and the card marker. However, `.filter(Boolean)` removes all falsy values — including that empty string — so the blank line is never emitted. When `fallback` is non-empty the output becomes `"<fallback>\n<!--adaptive-card-->..."` (single newline, no blank line). When `fallback` is empty (e.g., the card body contains only `Image` or `Input.*` elements that `generateFallbackText` doesn't handle), the output is just the raw `<!--adaptive-card-->` markers with no human-readable text at all — meaning channels that don't parse markers (Telegram, IRC, Slack) show nothing or raw comment syntax.
A more explicit construction avoids both problems:
```suggestion
const parts: string[] = [];
if (fallback) {
parts.push(fallback, "");
}
parts.push(`${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`);
if (templateData) {
parts.push(
`<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->`,
);
}
const markedText = parts.join("\n");
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 38696298. Replaced filter(Boolean) with explicit parts array construction -- blank-line separator is now preserved between fallback text and markers. Adopted the exact pattern from your suggestion.
| "", | ||
| "Card schema follows Adaptive Cards v1.5: https://adaptivecards.io/explorer/", | ||
| "Common body element types: TextBlock, ColumnSet, FactSet, Image,", | ||
| "Input.Text, Input.ChoiceSet, Input.Toggle, ProgressBar.", |
There was a problem hiding this comment.
ProgressBar is not part of the Adaptive Cards v1.5 specification
The official Adaptive Cards v1.5 schema explorer does not include a ProgressBar element — it was discussed in early preview proposals but was never added to the released specification. Including it in the tool description will cause the LLM to occasionally attempt to use it, producing cards that fail to render (or silently fall back to nothing) on compliant hosts.
The supported body element types for v1.5 are: TextBlock, RichTextBlock, Image, ImageSet, ColumnSet, Container, FactSet, Table, ActionSet, Media, Input.Text, Input.Number, Input.Date, Input.Time, Input.Toggle, Input.ChoiceSet.
| "Input.Text, Input.ChoiceSet, Input.Toggle, ProgressBar.", | |
| "Common body element types: TextBlock, ColumnSet, FactSet, Image,", | |
| "Input.Text, Input.ChoiceSet, Input.Toggle.", |
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/adaptive-cards/index.ts
Line: 35
Comment:
**`ProgressBar` is not part of the Adaptive Cards v1.5 specification**
The official [Adaptive Cards v1.5 schema explorer](https://adaptivecards.io/explorer/) does not include a `ProgressBar` element — it was discussed in early preview proposals but was never added to the released specification. Including it in the tool description will cause the LLM to occasionally attempt to use it, producing cards that fail to render (or silently fall back to nothing) on compliant hosts.
The supported body element types for v1.5 are: `TextBlock`, `RichTextBlock`, `Image`, `ImageSet`, `ColumnSet`, `Container`, `FactSet`, `Table`, `ActionSet`, `Media`, `Input.Text`, `Input.Number`, `Input.Date`, `Input.Time`, `Input.Toggle`, `Input.ChoiceSet`.
```suggestion
"Common body element types: TextBlock, ColumnSet, FactSet, Image,",
"Input.Text, Input.ChoiceSet, Input.Toggle.",
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Fixed in 38696298. Removed ProgressBar and updated the tool description to list the actual v1.5 spec elements: TextBlock, RichTextBlock, ColumnSet, Container, FactSet, Image, ImageSet, Table, ActionSet, Input.Text, Input.Number, Input.Date, Input.Time, Input.Toggle, Input.ChoiceSet.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c7451c67e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| .join("\n"); | ||
|
|
||
| return { | ||
| content: [{ type: "text" as const, text: markedText }], |
There was a problem hiding this comment.
Route Adaptive Card payload through channel card transport
The adaptive_card tool result is emitted as plain text with embedded markers, but no structured card field is passed to channel delivery. In the Teams channel, native card sending is only triggered when ctx.params.card is present (extensions/msteams/src/channel.ts), while plain text goes through sendText; this means invoking this tool in Teams will send marker-wrapped JSON text instead of an actual Adaptive Card attachment.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Acknowledged. This is intentionally scoped as a follow-up -- the Teams channel integration (detecting markers in msteams/outbound.ts and routing to sendAdaptiveCardMSTeams()) is called out in the PR description under "What's NOT in This PR". This PR adds the gateway-side tool only; the channel routing is a separate PR to keep scope focused per the repo's one-topic-per-PR rule.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 38696298a5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| // Command: /card for quick manual card testing | ||
| api.registerCommand({ | ||
| name: "card", |
There was a problem hiding this comment.
Rename /card command to avoid cross-plugin collisions
This plugin registers a /card command, but LINE already registers the same command name (extensions/line/src/card-command.ts), and plugin command registration rejects duplicates (src/plugins/commands.ts, registerPluginCommand). When both plugins are enabled, one handler is dropped based on load order, so either LINE’s existing card workflow or this extension’s test command becomes unavailable.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 323a2e9. Renamed /card to /acard to avoid the collision with LINE extension's /card command.
Adds an adaptive_card agent tool and /card command that enables the AI to respond with Adaptive Cards (v1.5) instead of plain text. Cards are embedded in tool result text using HTML comment markers so they flow through the existing pipeline without schema changes. Mobile apps (iOS/Android) extract the card JSON and render natively via SwiftUI/Jetpack Compose. MS Teams renders cards natively via Bot Framework. Channels that don't parse the markers see auto-generated fallback text.
- Remove unused stringEnum import (Greptile openclaw#1) - Replace filter(Boolean) with explicit parts construction to preserve blank-line separator between fallback text and markers (Greptile openclaw#2) - Remove ProgressBar from tool description (not in v1.5 spec), list actual supported elements (Greptile openclaw#3) - Guarantee non-empty fallback: add DEFAULT_FALLBACK constant used when generateFallbackText returns empty (Codex openclaw#2) - Extend generateFallbackText to handle Container, Image (altText), Table (cell text), and Input.* (label/placeholder) elements
3869629 to
c9e86bc
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9e86bc6e8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| case "RichTextBlock": | ||
| if (typeof el.text === "string") lines.push(el.text); |
There was a problem hiding this comment.
Extract RichTextBlock fallback from inlines
generateFallbackText handles RichTextBlock by reading el.text, but Adaptive Card RichTextBlock content is carried in inlines, not text. In practice, cards that use rich text lose their meaningful fallback on non-card channels and degrade to the generic default fallback, so users on Telegram/IRC/etc. miss the actual message content.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in d82c8a4. RichTextBlock now properly extracts text from inlines array (handles both plain string inlines and TextRun objects with .text property) instead of looking for a non-existent top-level text field.
|
Please make this as a third-party plugin that you maintain yourself in your own repo. Docs: https://docs.openclaw.ai/plugin. Feel free to open a PR after to add it to our community plugins page: https://docs.openclaw.ai/plugins/community |
|
Done! Moved to a standalone third-party plugin:
Will open a PR to add it to the community plugins page. Closing this PR since the code now lives in its own repo. |
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Summary
Adds an
adaptive-cardsgateway extension that gives the AI anadaptive_cardtool to respond with native Adaptive Cards (v1.5) instead of plain text on mobile and Teams.Why Adaptive Cards for OpenClaw
The Problem
OpenClaw's mobile chat is text-only. When the agent responds with structured content -- task status, options to choose from, data comparisons, forms -- it renders as a wall of markdown. On a phone screen, this means:
Why Adaptive Cards (Not Just A2UI)
OpenClaw already has A2UI (Google's Agentic UI) for the Canvas surface. A2UI is great for full-screen interactive experiences but it renders in a separate WebView, not inline in the chat stream.
Adaptive Cards solve a different problem: inline structured responses in the chat stream with native rendering.
The two are complementary: A2UI for rich canvas experiences, Adaptive Cards for inline chat interactions.
How It Works
Architecture
Marker Convention
Card JSON is embedded in the tool result text between HTML comment markers:
Mobile apps parse between
<!--adaptive-card-->and<!--/adaptive-card-->. Channels that don't understand the markers see only the fallback text above the markers.What's in This PR
extensions/adaptive-cards/index.ts(~245 lines)adaptive_cardtool -- The agent calls this to emit a card:{ "tool": "adaptive_card", "params": { "body": [ { "type": "TextBlock", "text": "Project Status", "weight": "Bolder" }, { "type": "FactSet", "facts": [ { "title": "Deploy API", "value": "Done" }, { "title": "Write tests", "value": "In Progress" } ]} ], "actions": [ { "type": "Action.Submit", "title": "Mark Complete", "data": { "task": "tests" } } ], "fallback_text": "Project: Deploy API (done), Write tests (in progress)." } }/acardcommand -- Quick testing (namedacardto avoid collision with LINE's/card):Auto-fallback generation -- When
fallback_textis omitted, generates plain text from card body (TextBlock, RichTextBlock, FactSet, ColumnSet, Container, Image altText, Table cells, Input labels).extensions/adaptive-cards/openclaw.plugin.jsonPlugin manifest. No configuration needed -- the extension is stateless.
What's NOT in This PR (Follow-ups)
This PR adds the gateway-side tool only. Mobile rendering requires follow-up PRs:
ChatMessageViews.swift, render viaAdaptiveCardViewAdaptiveCardViewmsteams/outbound.ts, route to existingsendAdaptiveCardMSTeams()adaptivecardsnpm package, parse markers in chat componentAction.Submittaps back to gateway as follow-up messagesReview Feedback Addressed
All review comments resolved across 4 commits:
stringEnumimport (dff459d)filter(Boolean)with explicitpartsarray to preserve blank-line separator (dff459d)ProgressBarfrom tool description (not in v1.5 spec) (dff459d)DEFAULT_FALLBACKconstant for empty card bodies (dff459d)generateFallbackTextfor Container, Image, Table, Input.* elements (dff459d)/card→/acardto avoid collision with LINE extension's/card(323a2e9)openclaw/plugin-sdk/coresub-path per lint rule (c9e86bc)acceptsArgs)Test Plan
pnpm checkpasses (oxlint + oxfmt + all custom lint rules)src/plugins/commands.test.tspasses (6 tests)src/secrets/target-registry.test.tspasses (3 tests)adaptive_cardtool when prompted for structured content/acard testemits a card with markerscc @steipete -- gateway foundation for native Adaptive Cards on mobile. Rendering side uses Teams-AdaptiveCards-Mobile (SwiftUI + Jetpack Compose, MIT licensed).
🤖 Generated with Claude Code