A2UI Flutter #

a2ui_flutter renders A2UI surfaces as Flutter widgets. In A2UI, an AI agent describes a user interface as JSON messages, and the app renders it with its own components. This package supports protocol v1.0 and v0.9 and includes the basic catalog of components.

To learn about A2UI, see a2ui.org.

Packages #

a2ui_flutter works with a2ui_core and a2ui_agent. Together they replace the genui package.

Getting Started #

Install #

flutter pub add a2ui_flutter a2ui_core

Render a surface #

A MessageProcessor from a2ui_core applies the agent's messages, and creates a surface for each createSurface message. An A2uiSurface renders one surface. This app renders a surface from two messages written by hand:

import 'package:a2ui_core/a2ui_core.dart';
import 'package:a2ui_flutter/a2ui_flutter.dart';
import 'package:a2ui_flutter/basic_catalog.dart';
import 'package:flutter/material.dart';

void main() => runApp(const MaterialApp(home: HelloA2ui()));

class HelloA2ui extends StatefulWidget {
  const HelloA2ui({super.key});

  @override
  State<HelloA2ui> createState() => _HelloA2uiState();
}

class _HelloA2uiState extends State<HelloA2ui> {
  final processor = MessageProcessor<ComponentImplementation>(
    catalogs: [basicCatalog()],
  );
  SurfaceModel<ComponentImplementation>? surface;

  @override
  void initState() {
    super.initState();
    processor.model.onSurfaceCreated.subscribe(
      (created) => setState(() => surface = created),
    );
    processor.model.onAction.subscribe(
      (action) => debugPrint('Action: ${action.name}'),
    );

    // In an app, these messages come from an agent.
    processor.processMessages([
      {
        'version': '1.0',
        'createSurface': {
          'surfaceId': 'hello',
          'catalogId': BasicCatalog.v1_0Id,
        },
      },
      {
        'version': '1.0',
        'updateComponents': {
          'surfaceId': 'hello',
          'components': [
            {
              'id': 'root',
              'component': 'Column',
              'children': ['greeting', 'button'],
            },
            {'id': 'greeting', 'component': 'Text', 'text': 'Hello, A2UI!'},
            {
              'id': 'button',
              'component': 'Button',
              'child': 'label',
              'action': {
                'event': {'name': 'greet'},
              },
            },
            {'id': 'label', 'component': 'Text', 'text': 'Say hi'},
          ],
        },
      },
    ]);
  }

  @override
  void dispose() {
    processor.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    final SurfaceModel<ComponentImplementation>? surface = this.surface;
    return Scaffold(
      body: surface == null ? null : A2uiSurface(surface: surface),
    );
  }
}

updateComponents gives the components as a flat list. The one with id root is the top of the tree, and each component names its children by id.

Pressing the button prints its action. The basic components are Material widgets, so render surfaces under a MaterialApp and inside a Material, such as a Scaffold body.

Connect an agent #

In an app, the agent's messages arrive through a transport, and the surfaces' actions and errors go back to the agent. So do the function calls and answers the processor makes. In this sketch, send is the app's transport:

late final MessageProcessor<ComponentImplementation> processor;
final surfaces = ValueNotifier<List<SurfaceModel<ComponentImplementation>>>(
  const [],
);

void sendToAgent(RendererToAgentMessage message) => send(
  RendererToAgentMessagePayload.of(message).toJson(),
  metadata: {'a2uiRendererDataModel': ?processor.getRendererDataModel()},
);

processor = MessageProcessor<ComponentImplementation>(
  catalogs: [basicCatalog()],
  outboundListener: sendToAgent,
  unifiedOutbound: true,
);

processor.model.onSurfaceCreated.subscribe((surface) {
  surfaces.value = [...surfaces.value, surface];
});
processor.model.onSurfaceDeleted.subscribe((id) {
  surfaces.value = [
    for (final surface in surfaces.value)
      if (surface.id != id) surface,
  ];
});

void onAgentMessages(Object? json) {
  try {
    processor.processMessages(json);
  } on A2uiError catch (error) {
    log('${error.code}: ${error.message}');
  }
}

// In build:
ValueListenableBuilder(
  valueListenable: surfaces,
  builder: (context, surfaces, _) => Column(
    children: [
      for (final surface in surfaces)
        A2uiSurface(key: ObjectKey(surface), surface: surface),
    ],
  ),
);

// On shutdown:
processor.dispose();

• Advertise the catalogs by sending processor.getRendererCapabilities(CapabilitiesOptions(versions: [A2uiProtocolVersion.v1_0])).toJson() as a2uiRendererCapabilities in the transport metadata.
• Send processor.getRendererDataModel() as a2uiRendererDataModel in the transport metadata of each message to the agent, when it is not null.
• With unifiedOutbound: true, outboundListener receives each action message, each processing error message from processMessages, the rendererFunctionResponse answering each callRendererFunction, and each callAgentFunction the processor makes. Subscribe to processor.model.onError (or surface.onError) to report runtime expression and action errors (and ignore ErrorMessage in outboundListener if you forward onError to the agent, since processMessages errors are emitted to both).
• Key each surface's widget by ObjectKey(surface), and remove it when the surface is deleted.
• Error listeners may call setState directly.
• On macOS, network images and media need the com.apple.security.network.client entitlement.

Prompt a model from the app #

An app that calls a model itself, with no agent server in between, prompts it with a2ui_agent:

flutter pub add a2ui_agent

A2uiGenerator writes the part of the system prompt that teaches the model A2UI and the app's catalogs. A ResponseStream parses the model's reply into messages for the processor as the reply streams in. The example app does this with Gemini, in lib/chat_session.dart.

Rendering Markdown #

Models often write Markdown in Text. To render it, wrap the surface in an A2uiMarkdown with a Markdown package's widget, such as MarkdownBody from flutter_markdown_plus:

A2uiMarkdown(
  builder: (context, markdown, style) => MarkdownBody(data: markdown),
  child: A2uiSurface(surface: surface),
);

A Text that no A2uiMarkdown renders, which includes every heading and caption Text, shows its text without common Markdown markers. A body Text that is one heading line, such as ## Summary, also takes the style of that heading level.

Defining Custom Components #

A v1.0 surface draws on several catalogs, so put your components in a catalog of their own and register it next to the basic catalog. The agent names it in each component's catalogId:

final rating = ComponentImplementation(
  name: 'Rating',
  schema: Schema.object(
    properties: {
      'label': CommonSchemasV1.dynamicString,
      'value': CommonSchemasV1.dynamicNumber,
      'onChanged': Schema.fromMap({r'$ref': r'common_types.json#/$defs/Action'}),
    },
    required: ['value'],
  ),
  builder: (context, node, props, buildChild) {
    final WritableBinding<Object?>? value = props.writable('value');
    return StarRating(
      label: props.string('label') ?? '',
      value: props.number('value') ?? 0,
      onChanged: value == null
          ? null
          : (stars) {
              try {
                value.set(stars);
                unawaited(props.action('onChanged')?.call());
              } on A2uiDataError catch (error) {
                log('${error.code}: ${error.message}');
              }
            },
    );
  },
);

final demoCatalog = CatalogImplementation(
  id: 'https://example.com/demo/catalog.json',
  protocolVersion: A2uiProtocolVersion.v1_0,
  components: [rating],
);

final processor = MessageProcessor<ComponentImplementation>(
  catalogs: [basicCatalog(), demoCatalog],
);

Declare dynamic values with CommonSchemasV1, and other shared types, such as Action above, with a $ref to common_types.json. Declare child references (ComponentId, ChildList) as top-level properties. Advertise the catalog by its id. An agent that does not use a2ui_core can be given its document, jsonEncode(demoCatalog.toJson()).

Writing a builder #

A builder receives the node, its resolved properties as ComponentProps, and a ChildWidgetBuilder:

• Read a value with props.string, number, boolean or value, whether the agent gave a literal or a binding.
• Write a value back through props.writable(key), which is null unless the property is bound to a data path. Its set throws A2uiDataError for a path the data model cannot write.
• Run an action with props.action(key). Each call runs as a user activation, which a function such as openUrl needs, so call it only from a gesture handler, synchronously.
• Enable a control only while its checks pass, with props.isValid, and show props.validationErrors. props.validationResults also holds the failing checks of warning and info severity, which leave the control valid. Its ValidationResult is a2ui_core's, so hide json_schema_builder's type of the same name.
• Render children with buildChild. A builder that wraps each child puts KeyedSubtree(key: NodeKey(child.instanceId), ...) outermost, so children keep their state when the list changes.
• Expect an unbounded height or width, as in a scroll view. Give a widget that needs a bound, such as a TextField, one with a LimitedBox.

Basic Catalog Components #

package:a2ui_flutter/basic_catalog.dart implements the v1.0 basic catalog with Material widgets. basicCatalog() builds all of it, and BasicComponents holds each component for a catalog of your own.

• Layout: Row, Column, List, Card, Tabs, Modal, Divider
• Content: Text, Image, Icon, Video, AudioPlayer
• Input: Button, TextField, CheckBox, ChoicePicker, Slider, DateTimeInput

The functions format for the locale passed to basicCatalog(), en-US by default. Inputs other than Button show a failing check of warning or info severity as a hint under them. A component with an accessibility.live of polite or assertive is a live region, which announces changes to its own label: a Text's text, a TextField's label, a Slider's label and value, or the check message under a CheckBox, ChoicePicker or Slider, but not a Button's child text, a CheckBox's label or a TextField's value.

Not supported yet:

• Measuring a surface's intrinsic size, as IntrinsicWidth and an AlertDialog's content do. Give the surface a fixed width there, or use a Dialog.
• Leaving video_player out of an app. Video and AudioPlayer play through it.
• An explicit accessibility.label in place of the label a control takes from its own text. On a Button, CheckBox or Slider the accessibility attributes go on a node around the control, and on a TextField or Text an explicit label is joined to the widget's own label.

Security #

openUrl fails unless the app passes a callback, such as url_launcher's launchUrl: basicCatalog(openUrl: launchUrl). On v1.0 it runs only from an action, which the basic components run only from a tap, such as a Button's. On v0.9 a bound openUrl call runs whenever it resolves, so with a callback the agent can open an http, https, mailto or tel URL without the user acting. Pass one only for an agent you trust.

Protocol v0.9 #

package:a2ui_flutter/v0_9/basic_catalog.dart implements the v0.9 basic catalog under the same names, with the v0.9 differences: Text has the h1 to h5 variants, TextField checks validationRegexp, and the components take the surface theme's primaryColor as the Material theme's primary color.

A v0.9 surface has one catalog, so a custom v0.9 component goes in a catalog with the basic components and functions it needs, under your own id:

import 'package:a2ui_flutter/v0_9/basic_catalog.dart' as v0_9;

final demoCatalogV0_9 = CatalogImplementation(
  id: 'https://example.com/demo/v0_9/catalog.json',
  // ratingV0_9 is a component whose schema uses the v0.9 types.
  components: [...v0_9.BasicComponents.all, ratingV0_9],
  functions: BasicCatalog.v0_9().functions.values.toList(),
  themeSchema: v0_9.BasicComponents.api.themeSchema,
  definitions: v0_9.BasicComponents.api.definitions,
);

Declare a v0.9 component's dynamic values with CommonSchemas.

To serve both versions, register a catalog of each with one processor, which routes each message by its version. Messages to the agent then follow the version of the surface they concern, its protocolVersion. A v0.9 surface's actions and errors go in v0.9 envelopes. A v0.9 agent reads processor.getRendererDataModel(version: A2uiProtocolVersion.v0_9) as a2uiClientDataModel, and processor.getRendererCapabilities(CapabilitiesOptions(versions: [A2uiProtocolVersion.v0_9])).toJson() as a2uiClientCapabilities. Each version's capabilities list every registered catalog, whatever its protocol version. Once surfaces of both versions send their data model, getRendererDataModel() needs a version, so a v1.0 agent reads it with version: A2uiProtocolVersion.v1_0.

Example #

example is a chat app in which a Gemini model, prompted through a2ui_agent, answers with v1.0 surfaces, and each event action and error on them goes back to the model. e2e_test runs it against the live model.

example_v0_9 is the same app on v0.9, and e2e_test_v0_9 runs it against the live model.

Gallery #

a2ui_explorer is the gallery app. It renders the v0.9 and v1.0 basic catalog examples and steps through their messages, next to each surface's data model and a log of actions and errors.

Tests #

test/catalog/basic/examples_test.dart renders every v1.0 basic catalog example and every v0.9 one.