Skip to content

[Feature]: Inline agent widgets in web chat via client-capability-gated tools #101790

Description

@steipete

Summary

Add an agent tool that renders an inline visual widget (SVG or sandboxed HTML) in the web chat, plus the underlying capability it needs: tool availability conditioned on what the user's current client surface can render (client-capability-gated tools).

Problem to solve

Agents can only answer in text/markdown in the web chat. For explanations that are inherently visual — flow diagrams, small dashboards, comparisons — a rendered inline widget communicates far better. The Canvas plugin already provides sandboxed embed surfaces, but there is no first-class tool for "render this markup inline in the chat transcript," and no mechanism to offer such a tool only where it can actually render. Today every gateway client (web UI, iOS, Android, macOS, CLI) collapses to the same internal message channel at tool-assembly time, so a web-only tool would appear (and dead-end) on phones and terminals too.

Proposed solution

Two parts, one PR:

  1. Client-capability-gated tool availability (generic seam). Gateway clients can already declare capability flags at connect (GATEWAY_CLIENT_CAPS, currently only tool-events). Add an inline-widgets capability, have the web Control UI/webchat client declare it, stamp the connected client's caps into the per-run message context at chat.send dispatch, and thread them into tool assembly. Tools (core or plugin) get a declarative requiredClientCaps contract; the tool builder filters tools whose required caps the originating client did not declare. Channel-originated runs (Telegram etc.) have no gateway client and therefore no caps — such tools are simply absent there. Additive only: a new capability string, no protocol version bump, no new config keys.

  2. show_widget tool (Canvas plugin). A new tool owned by the Canvas plugin (which owns the embed surfaces and sandbox policy): the agent passes markup (<svg …> or an HTML fragment) plus a title; the tool validates and persists it as a hosted same-origin document under the existing Canvas host path and returns the existing canvas-preview handle that the web chat already renders as a sandboxed iframe. SVG input is wrapped in a minimal HTML document server-side, so the chat renderer needs no new sanitization profile. Bounds enforced tool-side (payload size cap, per-session widget cap with cleanup).

Security model reuses what exists: widgets render in sandboxed iframes without allow-same-origin under the operator-controlled embedSandbox policy tiers, same-origin hosting only, and the served widget document carries a restrictive CSP so widget content cannot reach the network or parent origin.

Alternatives considered

  • Inline sanitized SVG/HTML directly in the chat DOM — requires opening up the strict DOMPurify allowlist and creates a standing XSS review surface; the sandboxed-iframe path already exists and is stronger.
  • Reuse the canvas tool's existing actions — those drive a separate canvas/A2UI surface (node-scoped panels), not the chat transcript, and the tool is not surface-gated; overloading it muddies both contracts.
  • Gate by message channel instead of client capability — web and native apps share the internal channel, so channel gating would surface the tool to phones where nothing renders.

Impact

  • Affected: web Control UI users; agent authors who want visual output; plugin authors get a documented way to ship surface-conditional tools.
  • Severity: feature gap; visual explanations currently require hosting content externally or using the separate canvas panel.
  • Frequency: everyday chat usage on the web surface.
  • Consequence: text-only answers for content that is naturally visual.

Evidence/examples

  • Existing render pipeline: canvas content blocks → sandboxed iframe (ui/src/pages/chat/components/chat-tool-cards.ts, ui/src/lib/chat/tool-display.ts), sandbox tiers documented in docs/web/control-ui.md (Hosted embeds).
  • Existing capability handshake: packages/gateway-protocol/src/client-info.ts (GATEWAY_CLIENT_CAPS).
  • Existing per-surface tool gating precedent: message-provider tool policy (src/agents/agent-tools.message-provider-policy.ts).
  • Prior art: Claude's show_widget/artifacts (agent-authored markup rendered in a sandboxed, design-system-styled container).

Additional information

  • Tool assembly must stay deterministic for prompt caching; capability-filtered tools follow the existing ordering rules.
  • Native apps can adopt later by declaring the capability and rendering the same hosted widget URL (macOS already embeds Canvas via WKWebView).
  • Tests: capability filter unit tests via the tool-assembly seams; widget tool tests for validation/persistence/handle shape; web UI normalizer/render tests per existing canvas test patterns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.enhancementNew feature or requestimpact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.impact:ux-frictionUser-facing flow adds avoidable confusion or support burden without fully blocking progress.issue-rating: 🌊 off-meta tidepoolIssue quality rating does not apply to this item.maintainerMaintainer-authored PRmaturity:stableIssue affects a taxonomy feature currently scored M4/M5.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions