Whitepaper

Build Better Chatbots

Best Practices for Power Virtual Agents
 April Whitepaper 2
2020

01 /
Why best practices matter

02 /
Best practices for Power Virtual Agents chatbots
04 Build the right chatbot
05 Understand your goals
05 Focus on the most common user requests
06 Start small and scale up
06 Create the right chatbot content
07 Choose the right topics
07 Plan your conversation tree and topic dialogs
09 Create the right trigger phrases
10 Use flows and skills effectively
11 Make your chatbot smart and flexible
11 Make a good first impression
11 Return most answers within the chatbot itself
12 Use entities
14 Let users switch among topics
14 Implement a high-quality conversational style
15 Use chatbot analytics to drive continuous improvement
16 Analyze unengaged sessions
16 Analyze abandoned sessions
17 Analyze escalated sessions

03 /
What to do now
 Whitepaper 3

Why best practices matter

Microsoft Power Virtual Agents enables people of all technical skill levels to create applications
that interact with users through natural language. These chatbots can handle customer support,
help employees work with internal applications, and more.

Following best practices A great chatbot addresses an important business problem in a

helps create chatbots straightforward way; people want to use it. The best way to create a
people want great chatbot is to follow well-established best practices. Figure 1
to use.
shows a big-picture view of a chatbot created with Power
Virtual Agents.

Figure 1: A chatbot provides a natural language understanding model, directs

users to topics through trigger phrases, and optionally connects with other
applications using flows and/or skills.
As the figure shows, users typically interact with a chatbot via text exchanges in a chatbot
window. Everything a user types is fed into a natural language understanding (NLU) model
provided by Power Virtual Agents. This model was trained with a huge amount of diverse data,
and it’s optimized for the way people use language with chatbots.

Regardless of how a This model determines the user’s intent, then maps that to trigger
Power Virtual Agents phrases. Each trigger phrase is connected to a specific topic, which
chatbot is used, the same implements a set of interactions with the user about a specific area.
best practices
apply. Chatbots created with Power Virtual Agents can also perform
actions.
 Whitepaper 4

To do this, a topic can invoke either a Power Automate flow or a skill (which is just an action
created with Microsoft Bot Framework). Flows and skills can carry out tasks on their own, and
they can also access other applications as needed. For example, a chatbot providing customer
service might access an ordering application to find order status, while a chatbot that answers
employee questions could interact with internal applications to determine, say, how many
vacation days an employee has available. Whatever a chatbot’s purpose, you can use the same
best practices for creating it.

Best practices for Power Virtual Agents

chatbots
The best practices for creating chatbots span all of the components shown in Figure 1. It’s useful
to group them under a few broad goals:
• Build the right chatbot
• Create the right chatbot content
• Make your chatbot smart and flexible
• Use chatbot analytics to drive continuous improvement
What follows looks at each of these areas.

Build the right chatbot

Every chatbot you deploy is intended to solve some business problem. Yet your organization
probably has lots of situations in which a chatbot could be helpful. How do you decide which
chatbots to build? And where do you start?

Best practices for building the right chatbot include:

• Understand your goals
• Focus on the most common user requests
• Start small and scale up
 Whitepaper 5

Understand your goals

Every chatbot project has goals. Here are some common examples:

Make sure • You want to lower your customer support costs by creating a support
everyone on chatbot that handles simple, repetitive requests rather than relying on a
your team is more expensive human agent.
clear about
the goals of • You want to give employees access to information 24/7 through a chatbot
the project. rather than relying on people who are only available during the day.
• You want to improve user satisfaction by providing chatbots that shorten
what would otherwise be long wait times to talk with a human agent.

Whatever your goals, everybody on the project should be clear about what you are (and aren’t)
trying to achieve with your new chatbot.

Focus on the most common user requests

It’s easy to be too ambitious, especially with your first chatbots. But once you examine the user
requests for the domain your chatbot addresses, you’re likely to find that a relatively small
number of requests occur most often. For example, users might ask a hundred different
questions, but just 20 of them comprise 80% of the traffic.

Your chatbot should initially address only these most common requests. Once it’s operational,
you can think about expanding to address the long tail of less common user requests. Figure 2
illustrates this idea.
 Whitepaper 6

Figure 2: Create a chatbot that addresses the most common user requests.
Automating responses to the most common requests will deflect the most volume from your
human agents. This means that focusing on commonly occurring questions will likely give you
the maximum return on your investment.

Start small and scale up

Start with a chatbot Chatbots that provide real value usually connect with some backend
that requires one or applications. While you probably want to begin by creating topics that
two connections don’t require backend connections, even your first chatbot should
with other contain one or two topics that do.
applications.
These connections are an important part of the value Power Virtual
Agents provides, so you want to start building your integration muscle
right away.

And don’t be surprised if creating these, which probably means connecting to backend
application programming interfaces (APIs), requires interacting with your IT organization.

Similarly, you should start by exposing your chatbot through only one channel. A chatbot
providing customer service might be made available solely on your organization’s public
website, for example, while an internally focused chatbot aimed at helping employees might be
accessible only through Microsoft Teams. Over time, you might find that allowing access
through other channels also has value, but it’s best to start by focusing on a single channel.

Once your chatbot has proven itself, you can use other aspects of Power Virtual Agents to work
at scale. For example, Power Virtual Agents includes the ability to export and import your
chatbot from one environment to another as needed. You can create, say, a development
environment for building the chatbot, a staging environment for testing it, and a third
environment for production. Using this approach helps you make updates with less risk of
damaging the chatbot version that’s serving real users.

Create the right chatbot content

Everybody wants to create the best possible chatbot content for their users. Best practices for
doing this include these:
• Choose the right topics
• Plan your conversation tree and topic dialogs
• Create the right trigger phrases
• Use flows and skills effectively
 Whitepaper 7

Choose the right topics

Defining the best topics for your chatbot requires understanding the questions your users will
ask and the kind of information you’ll need to provide. An obvious place to start is by using any
existing FAQ information in your area. If there’s already helpful information available, why not
build on it?

Power Virtual Agents

To help do this, Power Virtual Agents includes a Suggest Topics
can automatically feature. You supply URLs, then let Power Virtual Agents crawl these
create topics from webpages for FAQ information. The feature will automatically
FAQ pages. import suggestions that you can promote to preformatted topics,
including trigger phrases. You’re also free to edit or delete these
suggestions before they’re promoted to topics.

You’ll also need to create some topics yourself; working with an existing FAQ won’t be enough.
Things to keep in mind when doing this include the following:

• Think about topics from your user’s point of view. Never forget that they might know much
less about this area then you do. In fact, if the chatbot you create is helpful to you, it’s
probably too advanced to be a good fit for less-knowledgeable users.
• Create bite-size topics rather than a few large topics. Taking this approach makes triggering
more effective by clearly mapping trigger phrases to the specific topics that address
those areas.
• For larger groups, Power Virtual Agents provides a multi-author option that enables several
different people to work on the same chatbot simultaneously without stepping on
each other.

Plan your conversation tree and topic dialogs

Think hard about Chatbots typically interact with their users by asking questions. To
your question make your chatbot as effective as possible, you should carefully
hierarchy. think about what these questions should be and how a user will
navigate through them from the most general to the most specific.

To do this, think holistically about your chatbot’s overall conversation tree. What are the initial
questions a user asks to choose a particular topic? Once you’ve worked out what these should
be, define your topic dialogs, i.e., the hierarchy of questions asked inside each topic. Figure 3
illustrates these ideas.
 Whitepaper 8

Figure 3: A chatbot's conversation tree includes the topic dialogs from each of
the chatbot's topics.
For example, a retail chatbot might start by asking the user which of four things she wants to do:
find a store, place an order, check the status of an order, or return a purchased product. Her
answer could lead her into one of four topics, each with its own topic dialog.

Before doing any of this, you should gather existing information about what your users need.
For example, if you’re building a customer service chatbot, talk with existing service
representatives to learn what the most common questions are and the order in which they’re
typically asked.

Expect to create For simple interactions, you’ll create single-turn conversations with just
both single-turn and one question and one answer. All of the interactions generated from
multi-turn FAQs by the Power Virtual Agents Suggest Topics feature are single-
interactions. turn today. But more substantive topics require a multi-turn
conversation with multiple back-and-forth interactions between the
user and your chatbot.

For instance, if a user asks a retail chatbot whether there are any stores nearby, the chatbot
might respond with a question such as “What city do you live in” or “What is your postal code?”
to narrow down its possible responses. The user’s response to this will determine the chatbot’s
next response in the conversation.
 Whitepaper 9

Your goal is to understand what the user wants while still keeping the interaction simple, asking
as few questions as possible. It’s also a good idea to keep your responses short; strive to be both
precise and concise. Long paragraphs can be hard to read in a typical chatbot user interface.

Whatever the details of your chatbot interaction, you should carefully plan your question
hierarchy. The more thought you put in ahead of time, the better your chatbot will be.

Create the right trigger phrases

Good trigger phrases Writing good trigger phrases is probably the most important
are important. part of building a successful chatbot. As Figure 4 shows, Power
Virtual Agents makes it simple to define these.

Figure 4: Power Virtual Agents provides a visual tool for writing trigger phrases.
Start with five to ten trigger It’s best to start with between five and ten trigger phrases for
phrases for each each topic. As your chatbot gets used, you’ll learn more about
topic. how users express what they want. You can then add additional
trigger phrases over time to make your chatbot more useful.
 Whitepaper 10

Some general guidelines for writing good trigger phrases include these:
• Short trigger phrases, no more than ten words, are better.
• Trigger phrases for a particular topic should let users express what they want in multiple
ways. For example, a topic about finding a nearby store location might have trigger phrases
such as “Are there any stores around me”, “What’s the closest store”, and “I’d like to find a
nearby location”.
• Choose the words your users are likely to enter. Don’t default to the terminology used by
specialists in the area your chatbot addresses.
• Trigger phrases that lead to different topics should be semantically different. An ambiguous
trigger phrase that could lead to more than one topic is likely to require follow-up
questions to clarify what the user means, making your chatbot harder to use. If it’s too
difficult to differentiate the triggers for two different topics, consider combining those
topics and guiding your users with questions inside that combined topic.
• Try not to create trigger phrases with words that are common across multiple topics. The
Power Virtual Agents natural language understanding model can’t use these to differentiate
topics, so they add no value.

Creating good trigger Creating good trigger phrases is something of an art. Expect to iterate
phrases requires over time as your chatbot gets used, adding and changing these as
iteration. needed. Since modifying chatbots with Power Virtual Agents doesn’t
require specialized development skills, making these changes is
straightforward.

Use flows and skills effectively

Even though it’s best to start with chatbots that use only a couple of flows or skills, keep in mind
that Power Virtual Agents doesn’t limit you to just answering questions. Chatbots can also
perform actions by interacting with other applications and services, making them even more
useful (and more valuable to your organization).

Chatbots can Power Automate flows are the primary mechanism for connecting
connect to backend chatbots to backend applications. You can use many available connectors
applications using to interact with other software and services. If no connector exists for the
a flow. backend connection you need, you might choose to work with
professional software developers to create your own custom connector.
However you do it, flows provide a smooth link between your chatbot
and other software and services.
 Whitepaper 11

Chatbots can also access backend applications using a skill. Unlike a flow, however, which can be
created by people of all technical levels, a skill is created by professional developers. Still, if a
skill already exists that implements the connection you need, it makes sense to use it.

Make your chatbot smart and flexible

You want people to use your chatbot, which means making it both helpful and easy. Doing this
means building a chatbot that’s as smart and flexible as possible. Here are some of the most
important things you can do to achieve this goal:

• Make a good first impression

• Return most answers within the chatbot itself
• Use entities
• Let users switch among topics
• Implement a high-quality conversational style

Make a good first impression

First contact matters. Your chatbot’s initial greeting should clearly indicate what it can do. Here
are some examples:

• “I’m an order status chatbot. I can help you check the status of your order, change order
delivery options, and learn about why your order might have been delayed.”
• “I’m an airline scheduling chatbot. I can help you make or change reservations, check in for
a flight, and see whether your flight is on time.”
• “I’m a customer service chatbot. I can help with questions about your orders, store locations
and hours, and more.”
And remember: chatbots are not websites. Don’t make your initial greeting a menu.

A chatbot’s opening Along with describing what a chatbot does, its opening greeting also
greeting sets the sets the tone. Is this a support chatbot, which is likely to use a
tone for an professional voice? Or do you want your chatbot to have more
interaction. personality, perhaps something that aligns with your brand? Whatever
choice you make, give your user a consistent experience, starting with
the opening greeting.

Return most answers within the chatbot itself

Chatbots exist to help their users, which typically means returning answers. To do this effectively,
your chatbot should return most answers directly in its own user interface.
 Whitepaper 12

Power Virtual Agents provides a built-in Fallback topic for trigger phrases that match no topic,
and returning, say, Bing results for the user’s query can be acceptable here. This can be how you
handle the long tail of possible queries. But aim to create topics that answer at least the top 80%
of user queries within the chatbot itself.

Use entities
Entities let a chatbot A chatbot typically needs several pieces of information from a user to
acquire and use do its job. For example, a retail chatbot might need to know what item
information more someone is looking for, what color the item should be, and
easily. approximately how much it should cost. One way to acquire this
information is by asking three separate questions, one for each fact the
chatbot needs.

But Power Virtual Agents provides a better way to do this: your chatbot can use entities. This
feature lets your chatbot acquire and use information more easily—it reduces the number of
questions you need to ask.

To do this, you first choose which entities you want to use. Doing this is simple: as Figure 5
shows, you just select them from a list.
 Whitepaper 13

Figure 5: You choose the entities your chatbot should use by selecting them
from a list.
As the figure shows, Power Virtual Agents provides several built-in entities, including Age, Color,
Country, and Money. You can also define your own custom entities, such as ClothingItem in this
example.

Suppose you’ve chosen the entities ClothingItem, Color, and Money from the list above. Power
Virtual Agents will now automatically examine user inputs for information that matches these
entities, a process known as slot filling. For example, if you choose to use the built-in entity
Money in a conversation, the user might enter a value as “$100” or “a hundred dollars” or in
some other way, and the natural language model in Power Virtual Agents will figure out that this
is a monetary value of one hundred dollars. However values are entered, they’re stored in
variables, as Figure 6 illustrates.

Figure 6: Once you choose the entities you'd like to use, Power Virtual Agents
will automatically extract information that matches those entity types from user
responses.
In this example, your retail chatbot asks the user what they’d like to purchase (step 1). The user
responds with “I want to buy a black skirt for under $100” (step 2). Because you’re using the
entities ClothingItem, Color, and Money for this chatbot, Power Virtual Agents will extract these
 Whitepaper 14

three pieces of information from the response, storing this information in three different
variables (step 3). Your chatbot can then access these variables directly (step 4).

Using entities makes your life simpler by letting Power Virtual Agents more intelligently gather
information from user responses. More important, it makes life better for users, since entities
can find and store information from user responses, then make that information available later
in the conversation. Entities improve the user experience—they make your chatbot seem
smarter—and using them whenever possible is definitely a best practice.

Let users switch among topics

Chatbots should let People don’t always know what they want, and they often
users change their change their minds. Your chatbot should accommodate this by
minds. enabling the user to switch from one topic to another.

For example, a user might first ask why an order wasn’t delivered, then learn it was backordered,
then switch to asking why it was backordered. Your conversation tree should be structured to
allow this kind of typical human behavior.

Implement a high-quality conversational style

The better your chatbot, the more likely people are to use it. And the more people use it, the
greater the return on your chatbot investment. An important part of creating a great chatbot is
providing a high-quality conversational style.

There are many aspects to doing this. Some of the most important are these:
• Make the conversation move at a brisk pace. People get bored or frustrated easily, so not
wasting their time is critical.
• Be sure to verify that information is correct before doing something that’s hard to reverse,
such as placing an order.
• Do escalations well. Power Virtual Agents has a built-in Escalate topic that can pass an
existing conversation to a human agent. Even the best chatbots will sometimes frustrate
their users, so it’s important to do the handoff effectively.
• Always help the user know what your chatbot expects. Rather than asking an open-ending
question such as “What do you want to do?”, you might instead offer a list of options for
the user to choose from. Asking open-ended questions is also more likely to lead to
responses your chatbot can’t understand.
• If there’s going to be a delay, such as calling another application via a skill or a flow, tell the
user: “Please wait–I’ll be back in a few seconds.”
 Whitepaper 15

• End every conversation with the built-in topic End of Conversation. This topic asks the user
“Did this answer your question?”, letting you gather useful information about how your
chatbot is performing.

Use chatbot analytics to drive continuous improvement

Expect to invest One of the most important best practices is to continuously invest in your
in your chatbot chatbot throughout its life. If you don’t do this, the chatbot will become
throughout obsolete. A critical part of this is using Power Virtual Agents analytics to
its life. monitor your chatbot’s strengths and weaknesses. To do this, you must first
understand what the options are for each session, that is, each interaction
your chatbot has with a user. Figure 7 summarizes what can happen.

Figure 7: Sessions can be engaged or unengaged, and engaged sessions can end
in three different ways.
As the figure shows, sessions can be divided into two groups:
• Engaged sessions, in which a user-created topic is triggered.
• Unengaged sessions, in which no user-created topic is triggered. This means that the
question(s) the user asked didn’t match any of your trigger phrases.
For engaged sessions, there are three possible outcomes:
• The session can be resolved. For this to happened, you must end the interaction with the
built-in topic End of Conversation. As mentioned earlier, this topic asks the user “Did this
answer your question?” If the user either responds with “Yes” or doesn’t respond at all, the
session is marked as resolved. It’s definitely a best practice to end every interaction with this
built-in topic.
 Whitepaper 16

• The session can be escalated. This means that the user asked to speak to a human agent.
• The session can be abandoned. This means that the session was neither resolved nor
escalated within 60 minutes after it started.
No matter how a session ends, Power Virtual Agents keeps a transcript of that session. You can
look at these transcripts, filtering for only those you’re interested in. For example, you can read
all the transcripts for escalated sessions to help understand where users are getting frustrated.
To improve your chatbot, the most important things to analyze are sessions that were
unengaged, escalated, or abandoned.

Analyze unengaged sessions

In an unengaged If a user query doesn’t match any topic in your chatbot, that session
session, a user query is unengaged. With a new chatbot, you should expect these to be
didn’t match any fairly common, since you’re not trying to cover every possible
of the chatbot’s question your users might have. Over time, though, unengaged
topics. sessions are an important guide to where your chatbot needs
improvement.

If the questions that cause this are off-topic—requests like “What time is it?”—you probably
don’t need to worry about them. If they’re relevant to your chatbot’s goals, however, questions
that cause unengaged sessions are one of your best sources for learning what new topics you
need to add.

It’s also possible that unengaged sessions are caused by problems with your trigger phrases.
Maybe users use language in their questions that you hadn’t expected, for instance, or ask
questions you didn’t anticipate that can be addressed by existing topics. Whatever the case,
these questions point directly to how you should change your trigger phrases. And whatever the
situation, expect to iterate: make the changes, see if they help, then make more changes as
needed.

Analyze abandoned sessions

In an abandoned session, the Looking at abandoned sessions, those in which your user
user didn’t complete a didn’t complete a useful interaction, can help you see which
useful interaction topics in your chatbot need improvement. A high abandon
with the chatbot.
rate might be caused by several things, such as these:

• Maybe the user’s question triggers some topic—the session isn’t unengaged—but doesn’t
reach the right topic to answer the question. Fixing this might require rewording or adding
 Whitepaper 17

new trigger phrases. You should also consider combining topics that turn out to have
significant overlap.
• Perhaps the conversation is too complex or too long, making it hard for the user to follow.
Simplifying your topic could make it more useful. If you’ve implemented a complete
business process in the chatbot, for example, might you find a simpler approach?
If many users drop off at the same point in an interaction, this probably indicates a problem.
Looking at transcripts of abandoned sessions is an effective way to determine this.

Analyze escalated sessions

In an escalated session, Working out what causes users to escalate to a human agent is
the user asked to talk essential to decreasing these escalations. A key part of this is
with a human determining where in your chatbot’s conversations the escalation is
agent. happening. Are there specific points in the interaction where users
become frustrated with the software and ask to speak to a person?

If so, look for ways to improve the clarity and correctness of these. Once again, the best way to
do this is by looking at transcripts of escalated sessions.

You also need to make sure that your chatbot can handle the most frequently asked queries
throughout its life. Doing this requires adding fresh topics and trigger phrases that keep up with
changes in your business. Just as you need to train and retrain human agents, you need to
update your chatbot as needed.

What to do now
Applying best Using best practices with Power Virtual Agents helps you achieve four goals:
practices makes • Build the right chatbot
your chatbots
more valuable. • Create the right chatbot content
• Make your chatbot smart and flexible
• Use chatbot analytics to drive continuous improvement

Applying these best practices makes your chatbot more valuable and improves your return on
investment. Adopting them just makes sense.
 Whitepaper 18

© 2020 Microsoft Corporation. All rights reserved. This document is provided "as-is." Information and views
expressed in this document, including URL and other Internet Web site references, may change without
notice. You bear the risk of using it.

Some examples are for illustration only and are fictitious. No real association is intended or inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft
product. You may copy and use this document for your internal, reference purposes. You may modify this
document for your internal, reference purposes.

Some information relates to pre-released product which may be substantially modified before it’s
commercially released. Microsoft makes no warranties, express or implied, with respect to the information
provided here.