Skip to content

Extensions: add adaptive-cards extension for native GenUI#33486

Closed
VikrantSingh01 wants to merge 5 commits intoopenclaw:mainfrom
VikrantSingh01:feat/adaptive-cards-extension
Closed

Extensions: add adaptive-cards extension for native GenUI#33486
VikrantSingh01 wants to merge 5 commits intoopenclaw:mainfrom
VikrantSingh01:feat/adaptive-cards-extension

Conversation

@VikrantSingh01
Copy link
Copy Markdown

@VikrantSingh01 VikrantSingh01 commented Mar 3, 2026

Summary

Adds an adaptive-cards gateway extension that gives the AI an adaptive_card tool to respond with native Adaptive Cards (v1.5) instead of plain text on mobile and Teams.

Why Adaptive Cards for OpenClaw

The Problem

OpenClaw's mobile chat is text-only. When the agent responds with structured content -- task status, options to choose from, data comparisons, forms -- it renders as a wall of markdown. On a phone screen, this means:

  • No interactivity -- user must type "option 2" instead of tapping a button
  • No visual structure -- progress bars, fact tables, choice lists are all plain text
  • No native feel -- every response looks the same regardless of content type

Why Adaptive Cards (Not Just A2UI)

OpenClaw already has A2UI (Google's Agentic UI) for the Canvas surface. A2UI is great for full-screen interactive experiences but it renders in a separate WebView, not inline in the chat stream.

Adaptive Cards solve a different problem: inline structured responses in the chat stream with native rendering.

A2UI (Canvas) Adaptive Cards (Chat)
Where Separate full-screen surface Inline in chat messages
Renderer WebView + Lit.js Native SwiftUI / Jetpack Compose
First render ~300-500ms ~50ms
Memory ~30-50MB (WebView process) ~5-10MB (native views)
Accessibility Partial (web a11y) Full VoiceOver/TalkBack
Teams Not supported Native Bot Framework rendering
Use case Dashboards, visualizations Status cards, forms, choices

The two are complementary: A2UI for rich canvas experiences, Adaptive Cards for inline chat interactions.

How It Works

Architecture

Agent decides structured content is appropriate
    │
    ▼
Agent calls adaptive_card tool with body + actions
    │
    ▼
Tool returns text with embedded card JSON (marker tags)
    │
    ├─→ iOS/Android app: extracts JSON, renders via native SDK
    ├─→ MS Teams: extracts JSON, sends as Bot Framework attachment
    ├─→ Web UI: extracts JSON, renders via adaptivecards.io
    └─→ Telegram/Slack/IRC: shows fallback text (cards invisible)

Marker Convention

Card JSON is embedded in the tool result text between HTML comment markers:

Here are your 3 tasks: Deploy API (done), Write tests (in progress)...

<!--adaptive-card-->{"type":"AdaptiveCard","version":"1.5","body":[...]}<!--/adaptive-card-->

Mobile apps parse between <!--adaptive-card--> and <!--/adaptive-card-->. Channels that don't understand the markers see only the fallback text above the markers.

What's in This PR

extensions/adaptive-cards/index.ts (~245 lines)

adaptive_card tool -- The agent calls this to emit a card:

{
  "tool": "adaptive_card",
  "params": {
    "body": [
      { "type": "TextBlock", "text": "Project Status", "weight": "Bolder" },
      { "type": "FactSet", "facts": [
        { "title": "Deploy API", "value": "Done" },
        { "title": "Write tests", "value": "In Progress" }
      ]}
    ],
    "actions": [
      { "type": "Action.Submit", "title": "Mark Complete", "data": { "task": "tests" } }
    ],
    "fallback_text": "Project: Deploy API (done), Write tests (in progress)."
  }
}

/acard command -- Quick testing (named acard to avoid collision with LINE's /card):

/acard test              -- Send a test card to verify rendering
/acard {"type":"..."}    -- Send custom card JSON

Auto-fallback generation -- When fallback_text is omitted, generates plain text from card body (TextBlock, RichTextBlock, FactSet, ColumnSet, Container, Image altText, Table cells, Input labels).

extensions/adaptive-cards/openclaw.plugin.json

Plugin manifest. No configuration needed -- the extension is stateless.

What's NOT in This PR (Follow-ups)

This PR adds the gateway-side tool only. Mobile rendering requires follow-up PRs:

Follow-up What Depends on
iOS rendering Add Teams-AdaptiveCards-Mobile as SwiftPM dependency, parse markers in ChatMessageViews.swift, render via AdaptiveCardView This PR
Android rendering Add ac-core + ac-rendering as Gradle dependencies, parse markers in chat composable, render via AdaptiveCardView This PR
Teams integration Detect markers in msteams/outbound.ts, route to existing sendAdaptiveCardMSTeams() This PR
Web UI rendering Add adaptivecards npm package, parse markers in chat component This PR
Action routing Route Action.Submit taps back to gateway as follow-up messages iOS/Android PRs

Review Feedback Addressed

All review comments resolved across 4 commits:

  • Removed unused stringEnum import (dff459d)
  • Replaced filter(Boolean) with explicit parts array to preserve blank-line separator (dff459d)
  • Removed ProgressBar from tool description (not in v1.5 spec) (dff459d)
  • Added DEFAULT_FALLBACK constant for empty card bodies (dff459d)
  • Extended generateFallbackText for Container, Image, Table, Input.* elements (dff459d)
  • Renamed /card/acard to avoid collision with LINE extension's /card (323a2e9)
  • Changed import to openclaw/plugin-sdk/core sub-path per lint rule (c9e86bc)
  • Rebased on latest main to pick up CI fixes (secretref markers, command test acceptsArgs)

Test Plan

  • Extension builds cleanly (no TypeScript errors)
  • pnpm check passes (oxlint + oxfmt + all custom lint rules)
  • src/plugins/commands.test.ts passes (6 tests)
  • src/secrets/target-registry.test.ts passes (3 tests)
  • Gateway builds and starts with extension loaded
  • iOS app builds and launches on simulator (iPhone 17 Pro)
  • Android app builds and launches on emulator (Google Pixel 10)
  • Agent uses adaptive_card tool when prompted for structured content
  • /acard test emits a card with markers
  • Fallback text generated correctly from card body
  • Card JSON survives gateway sanitization (verified by reading chat history)

cc @steipete -- gateway foundation for native Adaptive Cards on mobile. Rendering side uses Teams-AdaptiveCards-Mobile (SwiftUI + Jetpack Compose, MIT licensed).

🤖 Generated with Claude Code

@VikrantSingh01 VikrantSingh01 changed the title feat(extensions): add adaptive-cards extension for native mobile GenUI feat(extensions): add adaptive-cards extension for native GenUI Mar 3, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 3, 2026

Greptile Summary

This PR adds an adaptive-cards gateway extension that gives the AI an adaptive_card tool and /card command to embed native Adaptive Card JSON (v1.5) in tool result text using HTML comment markers. The overall architecture is sound — the marker-based embedding approach is a clever way to pass card data through the existing pipeline without any schema changes, and graceful degradation to fallback text for unsupported channels is a good design principle. Three concrete issues need attention before merging:

  • Unused import (stringEnum on line 3) will fail the oxlint pre-commit hook.
  • filter(Boolean) bug (lines 95–104) silently removes the intended blank-line separator between fallback text and the card marker. It also means any card body consisting entirely of Image, Input.*, or other non-extractable elements produces an empty fallback — channels like Telegram/IRC would receive only the raw <!--adaptive-card--> comment markers as plain text.
  • ProgressBar in tool description (line 35) references an element type that does not exist in the released Adaptive Cards v1.5 schema, which will cause the LLM to occasionally generate invalid card schemas that fail to render on compliant hosts.

Confidence Score: 3/5

  • Safe to merge after fixing the three identified issues; the extension is additive and does not affect any existing functionality.
  • The extension is purely additive (new files only, no changes to core gateway), so it carries low risk to existing functionality. However, the unused import will break the pre-commit oxlint hook, the filter(Boolean) bug produces incorrect output in two real-world scenarios (non-blank-line separator and empty fallback), and the ProgressBar description error will cause the LLM to emit invalid card schemas. These are concrete, reproducible bugs that need to be fixed before the feature works correctly.
  • extensions/adaptive-cards/index.ts — contains all three flagged issues.

Last reviewed commit: 9c7451c

Comment thread extensions/adaptive-cards/index.ts Outdated
@@ -0,0 +1,208 @@
import { Type } from "@sinclair/typebox";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk";
import { stringEnum } from "openclaw/plugin-sdk";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused import

stringEnum is imported but never referenced anywhere in the file. This will likely trigger an oxlint error (no-unused-vars / no-unused-imports) during the pre-commit hook.

Suggested change
import { stringEnum } from "openclaw/plugin-sdk";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk";
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/adaptive-cards/index.ts
Line: 3

Comment:
**Unused import**

`stringEnum` is imported but never referenced anywhere in the file. This will likely trigger an oxlint error (`no-unused-vars` / `no-unused-imports`) during the pre-commit hook.

```suggestion
import type { OpenClawPluginApi } from "openclaw/plugin-sdk";
```

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 38696298. Removed the unused stringEnum import.

Comment thread extensions/adaptive-cards/index.ts Outdated
Comment on lines +95 to +104
const markedText = [
fallback,
"",
`${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`,
templateData
? `<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->`
: "",
]
.filter(Boolean)
.join("\n");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filter(Boolean) silently drops the blank-line separator

The empty string "" on line 96 is intended to produce a blank line between the fallback text and the card marker. However, .filter(Boolean) removes all falsy values — including that empty string — so the blank line is never emitted. When fallback is non-empty the output becomes "<fallback>\n<!--adaptive-card-->..." (single newline, no blank line). When fallback is empty (e.g., the card body contains only Image or Input.* elements that generateFallbackText doesn't handle), the output is just the raw <!--adaptive-card--> markers with no human-readable text at all — meaning channels that don't parse markers (Telegram, IRC, Slack) show nothing or raw comment syntax.

A more explicit construction avoids both problems:

Suggested change
const markedText = [
fallback,
"",
`${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`,
templateData
? `<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->`
: "",
]
.filter(Boolean)
.join("\n");
const parts: string[] = [];
if (fallback) {
parts.push(fallback, "");
}
parts.push(`${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`);
if (templateData) {
parts.push(
`<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->`,
);
}
const markedText = parts.join("\n");
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/adaptive-cards/index.ts
Line: 95-104

Comment:
**`filter(Boolean)` silently drops the blank-line separator**

The empty string `""` on line 96 is intended to produce a blank line between the fallback text and the card marker. However, `.filter(Boolean)` removes all falsy values — including that empty string — so the blank line is never emitted. When `fallback` is non-empty the output becomes `"<fallback>\n<!--adaptive-card-->..."` (single newline, no blank line). When `fallback` is empty (e.g., the card body contains only `Image` or `Input.*` elements that `generateFallbackText` doesn't handle), the output is just the raw `<!--adaptive-card-->` markers with no human-readable text at all — meaning channels that don't parse markers (Telegram, IRC, Slack) show nothing or raw comment syntax.

A more explicit construction avoids both problems:

```suggestion
      const parts: string[] = [];
      if (fallback) {
        parts.push(fallback, "");
      }
      parts.push(`${CARD_OPEN_TAG}${cardJson}${CARD_CLOSE_TAG}`);
      if (templateData) {
        parts.push(
          `<!--adaptive-card-data-->${JSON.stringify(templateData)}<!--/adaptive-card-data-->`,
        );
      }
      const markedText = parts.join("\n");
```

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 38696298. Replaced filter(Boolean) with explicit parts array construction -- blank-line separator is now preserved between fallback text and markers. Adopted the exact pattern from your suggestion.

Comment thread extensions/adaptive-cards/index.ts Outdated
"",
"Card schema follows Adaptive Cards v1.5: https://adaptivecards.io/explorer/",
"Common body element types: TextBlock, ColumnSet, FactSet, Image,",
"Input.Text, Input.ChoiceSet, Input.Toggle, ProgressBar.",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ProgressBar is not part of the Adaptive Cards v1.5 specification

The official Adaptive Cards v1.5 schema explorer does not include a ProgressBar element — it was discussed in early preview proposals but was never added to the released specification. Including it in the tool description will cause the LLM to occasionally attempt to use it, producing cards that fail to render (or silently fall back to nothing) on compliant hosts.

The supported body element types for v1.5 are: TextBlock, RichTextBlock, Image, ImageSet, ColumnSet, Container, FactSet, Table, ActionSet, Media, Input.Text, Input.Number, Input.Date, Input.Time, Input.Toggle, Input.ChoiceSet.

Suggested change
"Input.Text, Input.ChoiceSet, Input.Toggle, ProgressBar.",
"Common body element types: TextBlock, ColumnSet, FactSet, Image,",
"Input.Text, Input.ChoiceSet, Input.Toggle.",
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/adaptive-cards/index.ts
Line: 35

Comment:
**`ProgressBar` is not part of the Adaptive Cards v1.5 specification**

The official [Adaptive Cards v1.5 schema explorer](https://adaptivecards.io/explorer/) does not include a `ProgressBar` element — it was discussed in early preview proposals but was never added to the released specification. Including it in the tool description will cause the LLM to occasionally attempt to use it, producing cards that fail to render (or silently fall back to nothing) on compliant hosts.

The supported body element types for v1.5 are: `TextBlock`, `RichTextBlock`, `Image`, `ImageSet`, `ColumnSet`, `Container`, `FactSet`, `Table`, `ActionSet`, `Media`, `Input.Text`, `Input.Number`, `Input.Date`, `Input.Time`, `Input.Toggle`, `Input.ChoiceSet`.

```suggestion
      "Common body element types: TextBlock, ColumnSet, FactSet, Image,",
      "Input.Text, Input.ChoiceSet, Input.Toggle.",
```

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 38696298. Removed ProgressBar and updated the tool description to list the actual v1.5 spec elements: TextBlock, RichTextBlock, ColumnSet, Container, FactSet, Image, ImageSet, Table, ActionSet, Input.Text, Input.Number, Input.Date, Input.Time, Input.Toggle, Input.ChoiceSet.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c7451c67e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

.join("\n");

return {
content: [{ type: "text" as const, text: markedText }],
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Route Adaptive Card payload through channel card transport

The adaptive_card tool result is emitted as plain text with embedded markers, but no structured card field is passed to channel delivery. In the Teams channel, native card sending is only triggered when ctx.params.card is present (extensions/msteams/src/channel.ts), while plain text goes through sendText; this means invoking this tool in Teams will send marker-wrapped JSON text instead of an actual Adaptive Card attachment.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. This is intentionally scoped as a follow-up -- the Teams channel integration (detecting markers in msteams/outbound.ts and routing to sendAdaptiveCardMSTeams()) is called out in the PR description under "What's NOT in This PR". This PR adds the gateway-side tool only; the channel routing is a separate PR to keep scope focused per the repo's one-topic-per-PR rule.

Comment thread extensions/adaptive-cards/index.ts Outdated
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38696298a5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread extensions/adaptive-cards/index.ts Outdated

// Command: /card for quick manual card testing
api.registerCommand({
name: "card",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Rename /card command to avoid cross-plugin collisions

This plugin registers a /card command, but LINE already registers the same command name (extensions/line/src/card-command.ts), and plugin command registration rejects duplicates (src/plugins/commands.ts, registerPluginCommand). When both plugins are enabled, one handler is dropped based on load order, so either LINE’s existing card workflow or this extension’s test command becomes unavailable.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 323a2e9. Renamed /card to /acard to avoid the collision with LINE extension's /card command.

@VikrantSingh01 VikrantSingh01 changed the title feat(extensions): add adaptive-cards extension for native GenUI Extensions: add adaptive-cards extension for native GenUI Mar 9, 2026
Adds an adaptive_card agent tool and /card command that enables the AI to
respond with Adaptive Cards (v1.5) instead of plain text. Cards are embedded
in tool result text using HTML comment markers so they flow through the
existing pipeline without schema changes.

Mobile apps (iOS/Android) extract the card JSON and render natively via
SwiftUI/Jetpack Compose. MS Teams renders cards natively via Bot Framework.
Channels that don't parse the markers see auto-generated fallback text.
- Remove unused stringEnum import (Greptile openclaw#1)
- Replace filter(Boolean) with explicit parts construction to preserve
  blank-line separator between fallback text and markers (Greptile openclaw#2)
- Remove ProgressBar from tool description (not in v1.5 spec), list
  actual supported elements (Greptile openclaw#3)
- Guarantee non-empty fallback: add DEFAULT_FALLBACK constant used when
  generateFallbackText returns empty (Codex openclaw#2)
- Extend generateFallbackText to handle Container, Image (altText),
  Table (cell text), and Input.* (label/placeholder) elements
@VikrantSingh01 VikrantSingh01 force-pushed the feat/adaptive-cards-extension branch from 3869629 to c9e86bc Compare March 9, 2026 22:06
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9e86bc6e8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread extensions/adaptive-cards/index.ts Outdated
Comment on lines +187 to +188
case "RichTextBlock":
if (typeof el.text === "string") lines.push(el.text);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Extract RichTextBlock fallback from inlines

generateFallbackText handles RichTextBlock by reading el.text, but Adaptive Card RichTextBlock content is carried in inlines, not text. In practice, cards that use rich text lose their meaningful fallback on non-card channels and degrade to the generic default fallback, so users on Telegram/IRC/etc. miss the actual message content.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d82c8a4. RichTextBlock now properly extracts text from inlines array (handles both plain string inlines and TextRun objects with .text property) instead of looking for a non-existent top-level text field.

@openclaw-barnacle
Copy link
Copy Markdown

Please make this as a third-party plugin that you maintain yourself in your own repo. Docs: https://docs.openclaw.ai/plugin. Feel free to open a PR after to add it to our community plugins page: https://docs.openclaw.ai/plugins/community

@VikrantSingh01
Copy link
Copy Markdown
Author

Done! Moved to a standalone third-party plugin:

Will open a PR to add it to the community plugins page. Closing this PR since the code now lives in its own repo.

VikrantSingh01 added a commit to VikrantSingh01/openclaw-adaptive-cards that referenced this pull request Mar 10, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
VikrantSingh01 added a commit to VikrantSingh01/openclaw-adaptive-cards that referenced this pull request Mar 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants