-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[Feature]: Inline agent widgets in web chat via client-capability-gated tools #101790
Copy link
Copy link
Closed
Closed
Copy link
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.enhancementNew feature or requestNew feature or requestimpact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.Security boundary, credential, authz, sandbox, or sensitive-data risk.impact:ux-frictionUser-facing flow adds avoidable confusion or support burden without fully blocking progress.User-facing flow adds avoidable confusion or support burden without fully blocking progress.issue-rating: 🌊 off-meta tidepoolIssue quality rating does not apply to this item.Issue quality rating does not apply to this item.maintainerMaintainer-authored PRMaintainer-authored PRmaturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.enhancementNew feature or requestNew feature or requestimpact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.Security boundary, credential, authz, sandbox, or sensitive-data risk.impact:ux-frictionUser-facing flow adds avoidable confusion or support burden without fully blocking progress.User-facing flow adds avoidable confusion or support burden without fully blocking progress.issue-rating: 🌊 off-meta tidepoolIssue quality rating does not apply to this item.Issue quality rating does not apply to this item.maintainerMaintainer-authored PRMaintainer-authored PRmaturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.
Type
Fields
Priority
None yet
Summary
Add an agent tool that renders an inline visual widget (SVG or sandboxed HTML) in the web chat, plus the underlying capability it needs: tool availability conditioned on what the user's current client surface can render (client-capability-gated tools).
Problem to solve
Agents can only answer in text/markdown in the web chat. For explanations that are inherently visual — flow diagrams, small dashboards, comparisons — a rendered inline widget communicates far better. The Canvas plugin already provides sandboxed embed surfaces, but there is no first-class tool for "render this markup inline in the chat transcript," and no mechanism to offer such a tool only where it can actually render. Today every gateway client (web UI, iOS, Android, macOS, CLI) collapses to the same internal message channel at tool-assembly time, so a web-only tool would appear (and dead-end) on phones and terminals too.
Proposed solution
Two parts, one PR:
Client-capability-gated tool availability (generic seam). Gateway clients can already declare capability flags at connect (
GATEWAY_CLIENT_CAPS, currently onlytool-events). Add aninline-widgetscapability, have the web Control UI/webchat client declare it, stamp the connected client's caps into the per-run message context atchat.senddispatch, and thread them into tool assembly. Tools (core or plugin) get a declarativerequiredClientCapscontract; the tool builder filters tools whose required caps the originating client did not declare. Channel-originated runs (Telegram etc.) have no gateway client and therefore no caps — such tools are simply absent there. Additive only: a new capability string, no protocol version bump, no new config keys.show_widgettool (Canvas plugin). A new tool owned by the Canvas plugin (which owns the embed surfaces and sandbox policy): the agent passes markup (<svg …>or an HTML fragment) plus a title; the tool validates and persists it as a hosted same-origin document under the existing Canvas host path and returns the existing canvas-preview handle that the web chat already renders as a sandboxed iframe. SVG input is wrapped in a minimal HTML document server-side, so the chat renderer needs no new sanitization profile. Bounds enforced tool-side (payload size cap, per-session widget cap with cleanup).Security model reuses what exists: widgets render in sandboxed iframes without
allow-same-originunder the operator-controlledembedSandboxpolicy tiers, same-origin hosting only, and the served widget document carries a restrictive CSP so widget content cannot reach the network or parent origin.Alternatives considered
canvastool's existing actions — those drive a separate canvas/A2UI surface (node-scoped panels), not the chat transcript, and the tool is not surface-gated; overloading it muddies both contracts.Impact
Evidence/examples
ui/src/pages/chat/components/chat-tool-cards.ts,ui/src/lib/chat/tool-display.ts), sandbox tiers documented indocs/web/control-ui.md(Hosted embeds).packages/gateway-protocol/src/client-info.ts(GATEWAY_CLIENT_CAPS).src/agents/agent-tools.message-provider-policy.ts).Additional information